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Overview 


In 2010, Hillsborough Community College (HCC), a large multicampus institution in Tampa, 
Florida, worked with MDRC to create the Mathematics Access Performance Scholarship (MAPS) 
program to help academically underprepared community college students succeed in developmental 
math. MAPS provides an incentive for low-income students referred to developmental math to take 
their math courses early and consecutively, get help with math in an on-campus Math Lab, and 
strive for passing grades or better, in exchange for a modest scholarship contingent on performance. 
MAPS is part of the national Performance-Based Scholarship (PBS) Demonstration, which is testing 
performance-based scholarships with varying student populations and program requirements. 

MAPS offered eligible students the opportunity to earn a performance-based scholarship of up to 
$1,800 ($600 per semester for three semesters) for successfully completing a three-course sequence 
— comprising Beginning Algebra, Intermediate Algebra, and a first-level college math course — 
within a period of three consecutive semesters plus a summer term. MAPS is the only program in 
the PBS Demonstration that provides an incentive for a specified course sequence. MDRC used a 
random assignment design to study the effects of MAPS compared with HCC’s standard services. 
Academic outcomes were tracked for all sample members for two years. 

Key Findings 

The findings at HCC, like those across the PBS Demonstration, are modest but positive. 

• More than 90 percent of program group students earned a MAPS award in the first 
semester, but scholarship receipt rates declined in the second and third semesters. Scholar- 
ship receipt rates decreased partly because fewer students reenrolled each semester, although 
rates declined even after accounting for decreased enrollment. 

• MAPS students were much more likely to visit a Math Lab than control group students. 

MAPS students were 38 percentage points more likely than control group students to visit a 
Math Lab in the first semester; a similar difference occurred in the second semester. MAPS stu- 
dents also spent more time in the Math Labs on average than did control group students. 

• MAPS helped move students further along in the math course sequence, and MAPS 
students also accumulated more credits overall — in part because of their greater pro- 
gress in math. By the end of two years, MAPS students were 1 1 percentage points more likely 
than control group students to complete a college-level math course or Intermediate Algebra as 
their highest-level math course (48.7 percent versus 37.8 percent). The program’s estimated ef- 
fect on total credit accumulation is statistically significant after one semester, after one year, and 
after one year and one semester. After two years, MAPS students had earned 1 .6 more total 
credits than their control group counterparts, although the increase of 7 percent is no longer sta- 
tistically significant. 

• The program had no discernible impact on students’ retention. Program group students 
were no more likely to reenroll in subsequent semesters than were control group students. 

A future report will present a synthesis of the final results from sites across the PBS Demonstration. 
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Preface 


Community colleges, which serve the postsecondary needs of more than one-third of college 
students in the United States today, are the primary providers of remediation for students who 
enter college academically underprepared. Developmental (or “remedial”) math is a particular 
challenge for community college students. In one study of more than 250,000 community 
college students, only 20 percent of students who were referred to developmental math passed a 
college-level math course — a course that is almost always required for graduation. 

The Mathematics Access Perfonnance Scholarship (MAPS) program, part of MDRC’s 
Perfonnance-Based Scholarship (PBS) Demonstration, serves low-income students who are 
referred to developmental math. The program provides an incentive for students to take their 
math courses early and consecutively, get help with math in an on-campus Math Lab, and strive 
for passing grades or better, in exchange for a modest grant contingent on perfonnance. The 
findings presented in this report, from MDRC’s evaluation of the MAPS program, indicate that 
MAPS led more students to visit a Math Lab and propelled them further in the math sequence, 
with a positive impact on students’ registration for and completion of a college- level math 
course. MAPS students also accumulated more credits overall. However, some academic 
outcomes remain unaffected, such as student retention from semester to semester. These modest 
but positive findings at HCC align with those found at other colleges in the PBS Demonstration, 
indicating that perfonnance-based scholarships can give students a small push in the right 
direction. 

In general, impacts for perfonnance-based scholarship programs are slightly more 
positive than for other scholarship programs found in the literature. This result may reflect the 
targeting of programs; on average, students in the PBS Demonstration have one or more risk 
factors for not completing college, such as low-income status, parenthood, or being older than 
traditional college age, which may contribute to the larger effect of the contingent grant on 
academic outcomes. Financial incentives may be especially relevant to these students. 
Scholarship providers may find that targeting their scholarships in purposeful ways, for instance 
by creating incentives for academic success or participation in student services, generates a 
greater return on investment than merit-based scholarship programs. 

A cross-site report on the PBS Demonstration will be published in 2015. This report 
will include longer-tenn follow-up for all colleges, including HCC, and will look at patterns and 
impacts across colleges to draw lessons from the larger demonstration. These findings will add 
to the growing body of knowledge about perfonnance-based scholarship models and the 
efficacy of incentives for improving academic success among low-income students. 

Gordon L. Berlin 

President, MDRC 
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Executive Summary 


Community colleges, which serve the postsecondary needs of more than one-third of college 
students in the United States today, 1 are also the primary providers of remediation for students 
who enter college academically underprepared. 2 According to the National Center for Education 
Statistics, more than two-thirds of entering community college students take at least one 
remedial course within six years of first enrollment. 3 Unfortunately, many students who enter a 
course of remediation struggle to complete that course, and only a third of students who take a 
remedial course in any subject earn a postsecondary degree. 4 Developmental math is a particu- 
lar challenge: in one study of more than 250,000 community college students, only 20 percent 
of students who were referred to developmental math passed a college-level math course — a 
course that is almost always required for graduation. 5 

To tackle the issue of developmental math success, Hillsborough Community College 
(HCC), a large multicampus institution in Tampa, Florida, worked with MDRC to create the 
Mathematics Access Perfonnance Scholarship (MAPS) program in 2010. MAPS is a perfor- 
mance-based scholarship program that provides an incentive for low-income students in 
developmental math to take their math courses early and consecutively, get help with math in an 
on-campus Math Lab, and to strive for passing grades or better, in exchange for a modest grant 
contingent on perfonnance. MAPS is part of the Perfonnance-Based Scholarship (PBS) 
Demonstration, a national demonstration project that is testing perfonnance-based scholarships 
in a variety of settings, with different populations and varying program requirements. MAPS is 


*U.S. Department of Education, Institute of Education Sciences, National Center for Education 
Statistics, “Total Fall Enrollment in Degree-Granting Institutions, by Control and Level of Institution: 
1963 Through 2010,” Digest of Education Statistics (2011): Table 199. 

2 This report uses the terms “developmental education” and “remediation” interchangeably. Although the 
terms may be interpreted somewhat differently, both refer to courses provided to students who are judged to be 
underprepared for college-level coursework. 

’These data are based on U.S. Department of Education, National Center for Education Statistics (NCES), 
Beginning Postsecondary Students data from a computation using the NCES QuickStats website; see 
http://nces.ed.gov/datalab/quickstats. 

4 Clifford Adelman, Principal Indicators of Student Academic Histories in Postsecondary’ Education, 
1972-2000 (Washington, DC: U.S. Department of Education, Institute of Education Sciences, 2004); U.S. 
Department of Education, National Center for Education Statistics, “Percentage of Students Graduating 
with an Associate’s Degree Within 3 Years of Entry from the 2-Year Degree-Granting Institutions Where 
the Students Started as Full-Time, First-Time Students,” United States Education Dashboard (Spring 
2010). Website: http://dashboard.ed.gov. 

5 Thomas Bailey, Dong Wook Jeong, and Sung-Woo Cho, “Referral, Enrollment, and Completion in De- 
velopmental Education Sequences in Community Colleges,” Economics of Education Review 29 (2010): 255- 
270. 


ES-1 



an important part of the PBS Demonstration as it is the only program that provides an incentive 
for a specified course sequence. 

After two years, the MAPS program has modestly helped students move further along 
the required math course sequence. 

MAPS Program Model 

MAPS offered students the opportunity to earn a performance-based scholarship of up to $1,800 
for successfully completing a three-course sequence within a period of three consecutive 
semesters plus a summer term: 

1. Beginning Algebra: a developmental math course 

2. Intermediate Algebra: a transitional course bridging developmental and 
college-level math classes 

3. First level of college math: College Algebra, Elementary Statistics, or Math 
for Liberal Arts, depending on student’s major 

The MAPS scholarship was awarded at two payment points each semester: 

1. Initial payment: $100 was paid for remaining enrolled in the specified 
math course as of the end of the add/drop period (typically two weeks into 
the semester). 

2. Final payment: $500 was paid for successfully completing the specified 
math course with a “C” or better (with the payment made after final grades 
had come in for the semester, typically one to two weeks after the last day of 
classes) and meeting Math Lab requirements as follows: 

• Beginning Algebra: minimum of five visits and five total hours over the 
course of the semester 

• Intermediate Algebra: minimum of three visits and three total hours 
over the course of the semester 

• College-level math course: no Math Lab requirement 
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In addition, students who received a grade of “B” or better in the specified math course 
received a bonus award at the end of the semester, in the form of a math textbook for the next 
course or a book voucher. 6 

Students who did not succeed in one of the three courses by earning a passing grade 
were still eligible to earn the final payment of the scholarship for that course if they reattempted 
and passed it within the program period. 7 Additionally, students received occasional reminders 
via e-mail from program staff about program requirements. 

Evaluation and Research Sample 

MDRC used a random assignment research design to estimate the effects of MAPS on academ- 
ic outcomes. The evaluation includes an implementation study, an impact study, and a cost- 
effectiveness analysis. 

HCC targeted its MAPS program to students who met the following criteria: 

1 . 18 years of age or older 

2. Low-income, defined as having an Expected Family Contribution (EFC) of 
up to 5,273 8 

3. In need of Beginning Algebra (the highest level of developmental math) 

Students who met all of the eligibility criteria and were interested in participating in the 
research study were randomly assigned either to a program group that was eligible to participate 
in MAPS or to a control group that received standard college services. 


Eligible students typically received a math textbook as their bonus award for the first two courses, and a 
voucher for the third course. 

7 Students who attempted a course, received the initial payment of $100, and did not pass at the end of the 
term were able to retake the course. During the retake, students would not be eligible to receive the initial 
payment for enrolling, but would be eligible to receive the final payment of $500 for meeting the Math Lab and 
grade requirement in the course. 

8 The Expected Family Contribution is a measure of a student’s and/or family’s ability to contribute toward 
the cost of college and is calculated according to a formula established by federal law. Elements of the formula 
may include the family’s taxed and untaxed income, assets, and benefits, such as unemployment and Social 
Security; the size of the family; and the number of family members enrolled in postsecondary education in the 
given year. 

At the time of this study, students with an EFC of 5,273 or less were eligible to receive a Pell Grant. 
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Implementation Findings 

The MAPS program was implemented at Hillsborough Community College, a two-year college 
with five campus locations, three satellite locations, and an active distance-learning community 
in Florida. The MAPS evaluation was conducted at two of HCC’s largest campuses, Dale 
Mabry and Brandon. Both campuses had significant course offerings and established support 
resources (such as Math Labs) in place at the time of the study’s inception to support the 
program, as well as the support of campus leadership necessary to implement the program. 

The key findings on the implementation of MAPS follow. 

• MAPS operated largely as designed. 

Program staff fulfilled their duties as expected, and scholarship payments and book bo- 
nuses were distributed with few errors. 

• More than 90 percent of students in the program group earned a MAPS 
award in the first semester, but rates of scholarship receipt declined to 
slightly less than 60 percent in the second semester and around 30 per- 
cent in the third semester. 

Scholarship receipt rates decreased partly as a result of fewer students reenrolling each 
semester: only around 80 percent and 60 percent of program group students reenrolled in the 
second and third semesters, respectively. However, even after accounting for students who did 
not reenroll at HCC, rates of scholarship receipt declined: more than 95 percent of program 
group students who were enrolled in the first semester received a MAPS payment, while only 
slightly more than 50 percent of students who were enrolled in the third semester did. 

• MAPS students were much more likely to visit the Brandon and Dale 
Mabry Math Labs than control group students. 

MAPS students were 38 percentage points more likely than control group students to 
visit the Math Labs in the first semester — 87 percent of program group students visited the 
Math Labs, compared with 49 percent of control group students. A similar difference occurred 
in the second semester. In both the first and second semesters, MAPS students also spent more 
time and made more visits to the Math Labs on average than control group students — more 
than doubling the average number of visits and average number of hours spent there by control 
group students. 
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The Performance-Based Scholarship Demonstration 
Figure ES.l 

Highest Level of Math Completed After Two Years 
Hillsborough Community College 



SOURCE: MDRC calculations from Hillsborough Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre -random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Academic Findings 

Academic outcomes were tracked for all sample members for two years. The key findings are: 

• MAPS helped move students further along in the math course sequence. 

As shown in Figure ES.l, compared with their control group counterparts, MAPS 
students were 11 percentage points more likely at the end of two years to complete a 
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college-level math course or Intermediate Algebra as their highest-level math course. This 
translates to an estimated average effect of 0.7 math credit, or a 15 percent increase in math 
credit accumulation (not shown in the figure). 

• There is evidence that MAPS improved students’ overall academic pro- 
gress as measured by total credit accumulation. This occurred, in part, 
because students made greater progress in mathematics as a result of 
MAPS. 

As shown in Figure ES.2, the program’s estimated effect on total credit accumulation, 
including both math credits and credits for courses other than math, is statistically significant 
after one semester, after one year, and after one year and one semester. After two years, students 
who were offered MAPS earned 1.6 more total credits than their control group counterparts, a 7 
percent increase, although this is no longer statistically significant. Taken together, this evidence 
suggests that MAPS improved students’ overall academic progress. 

Additionally, exploratory analyses suggest that MAPS may be more effective for finan- 
cially independent students and for students who entered the study more than a year after 
graduating from high school or receiving a General Educational Development (GED) certifi- 
cate. (The fonner group may be more likely to take responsibility for their own fiscal situation, 
while the latter may have been out of the math classroom for a while.) This finding will be 
strengthened if it is replicated at other colleges that implement perfonnance-based scholarship 
programs. 

• The program had no discernible impact on students’ retention. 

Over the first four semesters, MAPS had no statistically significant impact on students’ 
semester-to-semester retention. That is, program group students were no more likely to reenroll 
in subsequent semesters than control group students. 

• The cost-effectiveness findings on MAPS are mixed and vary by out- 
come. MAPS was able to lower the cost per college-level math course 
completion when compared with the usual college experience but did not 
lower the cost of math credits and total credits earned. 

The analysis shows that the $1,394 to $1,863 of additional investment in each pro- 
gram group member resulted in an increase of 5.6 percentage points in the likelihood of 
completing a college-level math course. This impact is large enough that when costs (direct 
program costs plus the cost of credits attempted by students over two years) are tied to the 
number of college-level math course completions, the program lowers the cost per outcome 
achieved in comparison with the cost per outcome of the usual college services without the 
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The Performance-Based Scholarship Demonstration 
Figure ES.2 

Total Credits Earned After Two Years 
Hillsborough Community College 



First semester First year First year and one semester First two years 


SOURCE: MDRC calculations from Flillsborough Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre -random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

program. Specifically, the cost per completion of a college-level math course for the program 
group is up to 5 percent less than the cost per college-level math course completion for the 
control group. The program also resulted in students’ earning 0.7 more math credit and 1.6 
more credits overall; these impacts were not large enough, however, to lower the cost per math 
or total credits earned. 
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Conclusions and Next Steps 

The MAPS program at HCC was designed to promote progress through a sequence of three 
math courses by creating incentives for enrollment, use of support services, and math perfor- 
mance. The findings that are reported here indicate that the program increased students’ use of 
Math Labs and moved students further along in the math sequence — demonstrating that a 
program like this one can have an impact on developmental math progress. There is also 
evidence of modest improvement on overall academic progress. At the end of two years, the 
MAPS group was ahead of the control group by 1.6 credits. 

The findings at HCC align with those at other colleges in MDRC’s Performance-Based 
Scholarship Demonstration. By and large, the colleges in the demonstration have found that 
perfonnance-based scholarships are able to improve some key academic outcomes, such as 
students’ meeting the targeted academic benchmarks and earning more credits. However, some 
academic outcomes remain mostly unaffected, such as students’ remaining in school for the 
next semester. The findings at HCC, and across the demonstration, are modest but positive, 
indicating that perfonnance-based scholarships can give students a small push in the right 
direction. 

A cross-college report on the PBS Demonstration will be published in 2015. That report 
will include longer-tenn follow-up for all colleges, including HCC, and will look at patterns and 
impacts across colleges to draw lessons from the larger demonstration. These findings will add 
to the growing body of knowledge about perfonnance-based scholarship models and the 
efficacy of incentives for improving academic success among low-income students. 
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Chapter 1 

Introduction 


Community colleges, which serve the postsecondary needs of more than one-third of college 
students in the United States today, 1 are also the primary providers of remediation for students 
who enter college academically underprepared. 2 According to the National Center for Educa- 
tion Statistics (NCES), more than two-thirds of entering community college students take at 
least one remedial course within six years of first enrollment. 3 Unfortunately, many students 
who enter a course of remediation struggle to complete that course, and only a third of 
students who take a remedial course in any subject earn a postsecondary degree. 4 Develop- 
mental math is a particular challenge for community college students. In one study of more 
than 250,000 community college students, only 20 percent of students who were referred to 
developmental math passed a college-level math course — a course that is almost always 
required for graduation. 5 

Administrators at Hillsborough Community College (HCC), a large multicampus insti- 
tution in Tampa, Florida, found that many of their students struggled to get through develop- 
mental math and pass the college-level math courses needed for graduation. They worried that 
math was presenting an insunnountable obstacle for many students. To address this problem, 
HCC worked with MDRC in 2010 to create the Mathematics Access Perfonnance Scholarship 
(MAPS) program. MAPS is a perfonnance-based scholarship program that provides an incen- 
tive for low-income students in developmental math to take their math courses early and 
consecutively, get math help in an on-campus Math Lab, and strive for passing grades or better 
in exchange for a modest grant contingent on perfonnance. The hope was that getting students 
through the developmental math sequence and a college-level math course would help them 
overcome the obstacle that math had been presenting and receive a credential. 

MDRC evaluated MAPS using a random assigmnent design (explained below), widely 
considered to be the gold standard in social science research. After two years, MDRC found that 


’U.S. Department of Education, National Center for Education Statistics (2011). 

2 This report uses the terms “developmental education’’ and “remediation’’ interchangeably. Although the 
terms may be interpreted somewhat differently, both refer to courses provided to students who are judged to be 
underprepared for college-level coursework. 

’Based on U.S. Department of Education, National Center for Education Statistics, Beginning Post- 
secondary Students data from a computation using the NCES QuickStats website 
(http://nces.ed.gov/datalab/quickstats). 

4 Adelman (2004). 

5 Bailey, Jeong, and Cho (2010). 
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MAPS has modestly helped students move further along the required math course sequence. 
This report covers the background of HCC’s MAPS program, the program’s implementation, 
the impacts of the program on students’ academic progress, and the cost-effectiveness of the 
program. 


National Performance-Based Scholarship Demonstration 

MAPS is part of a demonstration project that is testing various performance-based scholarship 
programs at eight institutions and one intermediary across the country. MDRC launched the 
Performance-Based Scholarship (PBS) Demonstration in 2008 to test an innovative strategy for 
addressing two policy objectives: increasing financial support available to low-income students 
and creating an incentive for such students to complete their courses and make more timely 
progress toward degrees. The idea is to provide a supplement to existing federal and state 
financial aid that is contingent on enrolling in a minimum number of credit hours and achieving 
passing grades, among other requirements. 

Supported by a consortium of funders, partners, and postsecondary institutions, the PBS 
Demonstration is based on positive findings that emerged from MDRC’s Opening Doors 
Demonstration in Louisiana, which showed that such an incentive-based scholarship program 
had a number of positive effects for students, including boosting students’ credit accumulation, 
grades, and persistence. While the Louisiana results were impressive, there was an open 
question about whether perfonnance-based scholarships would be effective in other college 
settings, for different target groups of students, or with different aid amounts. The national PBS 
Demonstration tests these variations. Box 1.1 highlights the key principles of perfonnance- 
based scholarships. 

The early results from other sites in the PBS Demonstration show modest but positive 
effects on important markers of academic progress, including increases in credits earned and an 
increase in the proportion of students who meet the scholarship grade and credit requirement. 6 
In some sites, the programs also reduced the amount of debt that students took on during the 
program semesters. The short-term results of the demonstration suggest that perfonnance-based 
scholarships can have an impact on some important markers of academic success. 


6 Patel, Richburg-Hayes, de la Campa, and Rudd (2013). 
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Box 1.1 


Key Principles of Performance-Based Scholarships 

The programs in the PBS Demonstration vary by target population, performance benchmarks, 
scholarship amounts, and the integration of student services (among other things). But all of 
the programs incoiporate the following key principles: 

• Awards are paid if students meet basic conditions regarding enrollment and grades in 
college courses. The awards thus act as incentives, rewarding behavior associated with ac- 
ademic success. 

• The scholarships are paid to students based on their academic performance in the current 
term, regardless of their performance in previous terms. This is unlike merit-based aid, for 
which students have to first qualify based on their high school performance (for example, 
high school grade point average) or grades from a previous college term. 

• To reinforce the incentive nature of these scholarships, they are paid directly to students 
rather than to institutions. Students can use the money to cover any expenses, including 
those that could derail continued attendance and success (for example, child care or trans- 
portation needs).* 

• Performance-based scholarships are designed as a supplement to Pell Grants and state aid 
to help meet the needs of low-income students. In other words, the intervention gives stu- 
dents more money to cover academic and living expenses, and can potentially reduce their 
dependence on loans. ' 

NOTES: *Financial aid that covers tuition and fees involves a transfer from the financial aid office to the 
university rather than a direct payment to the student. While a reduction in the amount owed to the 
university should theoretically mean the same to a student as a check, economic experimentalists and 
behaviorists have long appreciated the salience of actual, tangible cash in hand. 

'Financial aid regulations prohibit students from receiving financial aid in excess of their need. (Such 
excess aid is considered income.) In cases where students’ full cost of attendance is met by financial aid 
that has already been awarded, federal work-study or loans may be displaced by the performance-based 
scholarship. 


Creating a Developmental Math Intervention 
at Hillsborough Community College 

The Mathematics Access Performance Scholarship program originated from an analysis of 
student data by administrators at HCC. The administrators were concerned about the lack of 
success experienced by the majority of HCC students who tested into developmental mathemat- 
ics. HCC offers three math classes below the college level: College Prep Math, the first and 
lowest level of developmental math; Beginning Algebra, the second and highest level of 
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developmental math; and Intennediate Algebra, a transitional course between developmental 
and college- level math. 7 

The Associate in Arts degree at HCC requires students planning to transfer to a four- 
year institution to take two college-level math courses. This adds up to four math courses for 
students who start at the highest level of developmental math and five math courses for students 
who start at the lowest level of developmental math. According to HCC’s historical data, fewer 
than 10 percent of students who enrolled in Beginning Algebra in fall 2007 had completed one 
college- level math course by fall 2008. 8 With so few students advancing through a college-level 
math course, HCC administrators were motivated to work with MDRC to design an interven- 
tion to help more students get through their math requirements. 

To address low rates of completion of developmental math at HCC, administrators 
identified a number of factors that they believed significantly contributed to their students’ lack 
of progress — and that HCC could try to address. These factors included (1) students’ reluc- 
tance to enroll in math courses early; (2) students’ reluctance to take math courses continuously; 
and (3) students’ lack of exposure to and use of academic support services. 

HCC’s priorities are supported by research. Many of the students who fail to complete 
their assigned developmental sequence do so, in part, as a result of not starting the sequence in 
the first place. 9 Simply getting students into the sequence may have impacts on their success. 
HCC administrators also believed that encouraging students to take their sequenced courses 
continuously would help them complete the courses. Their intent was to help students avoid 
“learning loss,” or the loss of academic knowledge and skills during periods away from school, 
which is well documented in the education literature on kindergarten through twelfth grade (K- 
12) for long breaks like summer. 10 In this literature, learning loss has been shown to be even 
worse for students of color and other at-risk groups. 11 The same may hold true for postsecond- 
ary students: The longer the time that passes between sequential math courses, whether between 


’Developmental courses do not provide college-level credits required for graduation. Intermediate Alge- 
bra, a transitional course, provides students with college-level elective credits but does not provide college- 
level math credits. 

8 This calculation includes only students who passed Beginning Algebra in fall 2007, Intermediate Algebra 
in spring 2008, and a college-level math course in fall 2008. These requirements may have resulted in HCC’s 
historical calculation showing a lower proportion of students passing college-level math compared with the 
proportion of MAPS control group students shown in Chapter 4; the MAPS calculation did not have compara- 
bly strict requirements. 

9 Bailey, Jeong, and Cho (2010). 

10 Cooper (2003). 

n Rock, Pollack, and Weiss (2004). 


4 



high school and college or between college courses, HCC administrators reasoned, the less 
likely students would be to retain the key knowledge and skills they need. 

Additional math help in the form of supplemental instruction, practice, and tutoring has 
been identified as a key support for students in developmental math courses. 12 Tutors may be 
housed in colleges in one of two ways: in a stand-alone tutoring program or in a learning 
assistance center, which provides a number of supports for students’ learning. 13 Learning 
assistance centers offer an array of services for students in need of support, such as tutoring and 
computer-assisted instruction. 14 In developmental education specifically, both of these have 
been identified as promising practices for increasing completion rates among community 
college students. 15 HCC already offered students these services in a centralized location — the 
Math Lab — and wanted students to make better use of this resource. 

These factors are addressed by the MAPS design, which provides students with finan- 
cial incentives to take their math courses early, seek out math assistance, pass their courses, and 
immediately enroll in the next course in the sequence. 


MAPS Program Model 

MAPS was designed to connect developmental math students with HCC’s academic support 
services and to improve their perfonnance in math courses by creating incentives for making 
choices that lead to academic success. The program was also meant to provide financial relief to 
the low-income students it targeted. The program made no changes within the classroom and 
can be considered “light-touch,” given that the only interaction between program coordinators 
and students in the program consisted of occasional reminders via e-mail about program 
requirements. 

MAPS offered students the opportunity to earn a performance-based scholarship of up 
to $1,800 for the successful completion of a three-course sequence (as shown in Figure 1.1): 

1 . Beginning Algebra: a developmental math course 

2. Intermediate Algebra: a transitional course bridging developmental and 
college-level math classes 


12 Fullmer (2012); Council for the Advancement of Standards in Higher Education (2010); Rheinheimer, 
Grace-Odeleye, Francois, and Kusorgbor (2010); Perin (2004). 

13 Rutschow and Schneider (2011). 

14 Perin (2004); Stern (2001). 

15 Bassett and Frost (2010). 
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The Performance-Based Scholarship Demonstration 
Figure 1.1 

Hillsborough Community College Math Progression 


Sequence targeted by MAPS 


College Prep 



Beginning 

Intermediate | 

| 1 
College-Level | 

Math 



Algebra 

Algebra 

„ 1 Math (1) 

4 credits 



4 credits 

3 credits 

*1 r 

3 credits | 

. j 


f 

| College-Level 
Math (2) 

I 3 credits 


□ Noncollege credit 


LJ Elective credit only 


j College-level math credit 


SOURCE: Hillsborough Community College course catalog. 

NOTE: All sample members were eligible to enroll in Beginning Algebra at baseline. HCC students typically need two college-level courses to receive an 
Associate in Arts degree. The first college-level math course that students take may be College Algebra, Elementary Statistics, Topics in Mathematics, or 
Explorations in Mathematics. For their second college-level math course, students may take another of those four courses, or one of several more advanced 
courses not listed here. 





3. First level of college math: College Algebra, Elementary Statistics, or Math 
for Liberal Arts, depending on the student’s major 

The MAPS scholarship was awarded at two payment points each semester: 

1. Initial payment: $100 was paid for remaining enrolled in the math course as 
of the end of the add/drop period (typically two weeks into the semester). 

2. Final payment: $500 was paid for successfully completing the math course 
with a “C” or better (with the payment made after final grades had come in for 
the semester, typically one to two weeks after the last day of classes) and 
meeting Math Lab requirements as follows: 

• Beginning Algebra: minimum of five visits and five total hours over the course 
of the semester 16 

• Intermediate Algebra: minimum of three visits and three total hours over the 
course of the semester 

• College-level math course: no Math Lab requirement 

In addition, students who received a grade of “B” or better in the specified math course 
received a bonus award at the end of the semester, in the fonn of a math textbook for the next 
course or book voucher. 17 

The initial payment was designed to get money into students’ hands early so that they 
could spend it on early semester expenses, such as textbooks. The first payment also functioned 
to make the scholarship seem “real” to students, a tangible reminder of the program and what 
was available to them. Students earned this payment by still being enrolled in the proper math 
course through the end of the add/drop period. (For the fall and spring semesters, this date falls 
at the end of the first week of class; during the summer semester, this period is the second day 
of class.) 

The final payment was designed to offer an incentive for students to use the campus 
Math Labs and pass the math course in which they were enrolled. The grade benchmark for 


16 In setting these benchmarks, MDRC relied on lessons learned from evaluating two student success 
courses with required visits to “Success Centers” that operated at Chaffey College in 2005-2007. Student 
success courses teach students how to navigate the college environment through goal setting, study skills, 
college regulations, and so forth. See Scrivener, Sommo, and Collado (2009). 

17 Eligible students typically received a math textbook as their bonus award for the first two courses, and a 
voucher for the third course. 
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earning the scholarship was set at the passing rate of “C.” Additionally, in the first two courses 
in the sequence, students had to visit the Math Labs a minimum number of times and for a 
minimum number of hours in order to receive the final payment. HCC’s Math Labs are math- 
focused resource centers staffed by trained tutors that offer a range of activities for students, 
including one-on-one tutoring, group tutoring sessions, and online math instruction. The Math 
Labs and students’ usage are described in more detail in Chapter 3. 

The bonus award, given to students who not only pass their course but earn a grade of 
“A” or “B,” was designed to motivate students to aim higher. Earning a textbook for the next 
math course was intended to further motivate students to continue on in the sequence in the 
following semester, since the cost of math textbooks (approximately $150 to $200) can be 
prohibitive. Some students may not buy books at all, which puts them at an academic disad- 
vantage. Having the cost of a textbook covered before the semester starts was theorized to be a 
factor that would motivate students to enroll in the next course. 

MAPS was designed to encourage students to complete the three-course sequence with- 
in a period of three semesters and one summer semester (referred to in this report as the “study 
period”). The modest flexibility implicit in this four-semester time frame was intended to be 
responsive to the realities of students’ lives (for instance, students may need to work or care for 
their children over the summer), while still encouraging persistence in mathematics. In addition, 
students who did not succeed in one of the three courses by earning a passing grade were still 
eligible to earn the final payment of the scholarship for that course if they reattempted it and 
passed. Multiple chances at the course and the final scholarship payments were intended to help 
motivate students to reenroll and persist in mathematics. 18 

MAPS was designed so that students were paid directly, receiving their payments on 
bank cards that they could use on campus or take to a bank to withdraw cash. 19 This is a core 
element of MDRC’s performance-based scholarship model. Paying the scholarships directly to 
students allows them to use the funds at their discretion. Students can choose for themselves 
which expenses are most pressing, whether they be educational, such as textbooks or tutoring, 
or personal, such as transportation or child care. All these expenses can be barriers to successful 
academic progress or completion. 


18 Students who attempted a course, received the initial payment of $100, and did not pass at the end of the 
term were able to retake the course. During the retake, the student would not be eligible to receive the initial 
payment for enrolling, but would be eligible to receive the final payment of $500 for meeting the Math Lab and 
grade requirement in the course. The maximum scholarship award per course was $600, regardless of the 
number of attempts a student made for each course. 

19 These cards, called “HOC One” cards, are used to disburse all financial aid refunds to students at HOC. 



Theory of Change 

Why might performance-based scholarships combined with required Math Lab visits be 
expected to lead to better outcomes for students? A logic model depicting the theory of change 
for the MAPS program is shown in Figure 1.2. Drawing on behavioral economics and discus- 
sions with financial aid experts, MDRC postulated that performance-based scholarships might 
influence students’ thinking and behavior in several ways. 20 First, the benchmarks required to 
earn the money may motivate students to enroll in math immediately and continuously, make 
use of the Math Labs to get the extra support they need, and work harder at earning good 
grades. Second, receipt of a scholarship may help students to keep employment to a reasonable 
level and to devote more time to school. Third, receipt of a scholarship may help students cover 
necessary expenses associated with going to college — from books and supplies to transporta- 
tion — and thereby increase their odds of success. 

If students respond to the scholarships in the ways outlined above, they are expected to 
meet the conditions of the performance-based scholarship, feel increased confidence in their 
ability to succeed, and feel lower levels of stress about money while in college. They are also 
expected to have reduced or eliminated the math coursework required to earn a credential. 
These short-term outcomes may lead to greater persistence in college and a shorter time to earn 
a degree. Ultimately, these behavioral changes are postulated to lead to higher graduation rates. 
Performance-based scholarships are predicated on the idea that relatively modest supplements 
to financial aid can provide needed help and lead to better academic outcomes. 


MDRC’s Evaluation of MAPS 

MDRC’s evaluation of MAPS uses a random assignment design to estimate the effects of the 
program on service usage and academic outcomes. Widely considered to be the gold standard in 
social science research, random assignment results in this context in two groups of students who 
are similar at the outset of the study in regard to both their observable characteristics (race, 
gender) and unobservable ones (motivation, personal obstacles). As a result, any subsequent 
differences in (or “impacts” on) outcomes between the two groups can be confidently attributed 
to MAPS. 

There are limits to this design, however. MAPS involves multiple components offered 
as a package to program group students. This study examines whether that package affects 


20 MDRC used guidance from the large literature on stereotype threat (Steele and Aronson, 1995) and 
discussions with noted cognitive psychologist Edward Deci on motivation in developing the conceptual 
framework. 
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The Performance-Based Scholarship Demonstration 


Figure 1.2 

Theory of Change for the Mathematics Access Performance Scholarship Program 

Hillsborough Community College 


Program 

Components Mediators Outcomes 


Scholarship offer 

• $600 per course for three- 
course sequence 

• Two payment points 

• Grade and Math Lab 
requirements 

• Textbook awarded for 
earning an "A" or "B" 



Academic 

• Increased knowledge of math 
support services for students 

• More effective study time 

• Improved mastery of material 

• Better choices made about 
math enrollment 


Math Lab visits 

• Minimum number of 
visits to and hours in 
Math Lab 

► 



Financial 

• Financial incentive to 
prioritize study time 

• Scholarship receipt 

• Reduced financial stress 

• More money to cover 
expenses 


Communications 

• E-mail reminders about 
program requirements 



Short Term 
(1-2 years) 


Longer Term 
(2+ years) 


Academic progress 


Academic progress and 

• Progress through 


attainment 

math sequence 


• Persist in college 

• Earn more credits 


• Earn more credits 

• Persist in college 


• Earn a degree or certificate 

• Transfer to a four-year 
institution 





student outcomes, but it cannot disentangle the impacts of individual components such as Math 
Lab visits or scholarship payments. Rather, the evaluation produces a reliable estimation of the 
impact of the program as a whole. 

The evaluation seeks to answer the following key research questions: 

• How was MAPS implemented and was it implemented as designed? 

• Did the program increase students’ Math Lab usage and scholarship receipt? 

• Does the opportunity to participate in MAPS improve students’ academic 
progress? 

• What does MAPS cost? Is it a cost-effective strategy to achieve the desired 
academic outcomes? 


The Remainder of the Report 

This report discusses the program’s implementation and its impacts on student achievement. 
Chapter 2 provides more detail on the HCC campus, as well as information about the random 
assignment process, data sources used in the evaluation, and baseline characteristics of the 
sample. Chapter 3 reports on the implementation of MAPS, student outcomes regarding service 
use, and scholarship receipt. Chapter 4 examines the academic impacts of the program. Chapter 
5 explores the costs and cost-effectiveness of the program. Chapter 6 concludes the report by 
summarizing findings and discussing where the field is heading in terms of developmental 
education reform. 
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Chapter 2 

Background and Data Sources 


MDRC’s evaluation of the Mathematics Access Perfonnance Scholarship (MAPS) program 
uses a random assignment design to test the effectiveness of a performance-based scholarship in 
improving perfonnance in a math course sequence. This chapter provides infonnation about the 
college and the target population, discusses the random assignment process, presents character- 
istics of the study sample members, and outlines the data sources used in the report for the 
implementation, impact, and cost analyses. 


The College 

The MAPS program was implemented at Hillsborough Community College (HCC) in Florida. 
Located in the greater Tampa metropolitan area, HCC is a two-year college with five campus 
locations, three satellite locations, and an active distance-learning community (students who are 
studying remotely). Among the larger community colleges in the state of Florida, HCC serves a 
diverse population of more than 46,000 students throughout Hillsborough County each year. 

The MAPS evaluation was conducted at two of HCC’s largest campuses, Dale Mabry 
and Brandon. The administration at HCC wanted MAPS to reach a significant portion of the 
college population while keeping the program centrally located. Both campuses had significant 
course offerings and established resources (such as Math Labs) in place at the time of the 
study’s inception; they also had the support of campus leadership for the program. 

Dale Mabry Campus is the college’s oldest and largest campus, located in the urban 
community of Drew Park in the heart of Tampa. This campus is home to nearly half of HCC’s 
student population. Career and technical programs in health, computer science, business, 
hospitality and tourism management, and culinary arts are located at Dale Mabry. Additionally, 
it is home to many of the college’s student services and resources, including HCC’s Dental 
Clinic, intercollegiate athletic programs, and student housing facilities. 

Brandon Campus is located in Brandon, a suburb of Tampa, and primarily serves the 
central portion of Hillsborough County. Though the campus has broad course offerings within 
both Associate of Arts and Associate of Science programs, it excels in science, technology, 
engineering, and mathematics (STEM) curricula. Among HCC’s campuses, Brandon is posi- 
tioned as the center for technology, research, and science education. 
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The Target Population 

The study at HCC targeted both new and continuing students in need of the highest level of 
developmental math, Beginning Algebra. In order to be eligible for the MAPS study, students 
were required to be: 

• 18 years of age or older 

• Low-income, defined as having an Expected Family Contribution 1 (EFC) of 
5,273 or less 

• In need of Beginning Algebra, which included the following three groups of 
students: 

o Students who had successfully completed the first level of developmental 
math, College Prep Math, and were ready to move on to Beginning Algebra 

o Students who placed directly into Beginning Algebra 

o Students who had not successfully completed Beginning Algebra and were 
required to retake the course 

These requirements were chosen specifically to reach a wide range of students believed 
by college administrators to be at high risk for not completing the developmental math se- 
quence. Administrators hoped that, with MAPS, the students could potentially complete the 
three-course sequence within three semesters and one summer term, and that the scholarship 
could help them with the cost of postsecondary education. 


Recruitment and Random Assignment 

MAPS operated at HCC from fall 2010 through fall 2012. The study sample was recruited over 
three time periods — before the start of the fall 2010, spring 2011, and fall 2011 semesters — 
producing three cohorts of students. The program period lasted three semesters plus one 
summer semester for each cohort. A total of 1,075 students were randomly assigned into the 
study: 336 students in the fall 2010 cohort, 327 students in the spring 2011 cohort, and 412 


The EFC is a measure of a student’s or family’s ability to contribute to the cost of college and is calculat- 
ed according to a formula established by federal law. Elements of the formula may include the family’s taxed 
and untaxed income, assets, and benefits, such as unemployment and Social Security; the size of the family; 
and the number of family members enrolled in postsecondary education in a given year. 
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students in the fall 2011 cohort. Sixty-two percent of students were randomly assigned to the 
program group and 38 percent were assigned to the control group. 2 

HCC staff in the office of institutional research identified eligible students using a data- 
base housed in their department. Eligible students received letters by postal mail and e-mail 
inviting them to an information session about the program and requesting a reply. The invitation 
letter briefly described the program and informed students that, because there were a limited 
number of slots, students who came to the information session would be selected by lottery to 
enroll in the program. 

HCC held multiple information sessions during each round of random assignment. Dur- 
ing these sessions, college staff explained how MAPS worked, what was required of students, 
and the purpose of the study. Students who wished to participate in the study were asked to sign 
an “in formed consent” form detailing the terms of their participation in the study, including a 
description of the data that would be collected and analyzed. Before they were randomly 
assigned, students were also required to fill out a baseline information form (BIF), which asked 
for background information such as students’ age, race and ethnicity, marital status, prior 
educational experience, and current financial and employment situation. Students were notified 
of their research group status immediately after filling out the informed consent form and the 
BIF. If students were not already registered for Beginning Algebra, they had the option to 
register for a class at that time. 

Since these courses are often oversubscribed, HCC administration reserved a number 
of seats in Beginning Algebra courses at both Dale Mabry and Brandon campuses for study 
participants to ensure that students who enrolled in the study were able to take the first math 
course in the sequence. Both program and control group students were able to take advantage 
of the reserved seats; as a result, reserved seats are not part of the intervention being studied. 
As students were recruited into the study, staff found that most students who came to infor- 
mation sessions and showed interest in joining the study had already registered for Beginning 
Algebra. As the start of the semester neared, the reserved seats that were unused were re- 
leased gradually to the general student body to ensure that students in need of the course 
outside of the study were able to register. For the second and third courses in the sequence, no 
seats were held for students. 


2 A random assignment ratio of 3:2 was used. 
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Sample Characteristics 

Table 2.1 presents some demographic and other background characteristics for the full MAPS 
study sample. (See Appendix Table A.1 for a comparison of program and control group students.) 

The majority of sample members, roughly 66 percent, are female. The sample compris- 
es white, black, and Hispanic students equally; there is no racial majority. The average age of 
the sample at the time of random assignment was 27 years, reflecting the fact that the majority 
of students were not of traditional college age. The majority of students were unmarried, and 
around 44 percent had at least one child. Nearly 40 percent of students were receiving some 
fonn of government benefits. About half the students reported being employed at the time of 
random assignment, and of those, roughly 70 percent were working more than 20 hours per 
week. The majority of students intended to attend HCC full time (12 or more credits) in their 
first semester of the program. One-third of the students are the first in their families to attend 
college and about 25 percent reported speaking a language other than English regularly at home. 

To assess the representativeness of the sample, Table 2.2 presents background and de- 
mographic characteristics for the study sample compared with the general HCC population and 
the MAPS-eligible population. 3 The data show that MAPS-eligible students include a higher 
proportion of females than does the general HCC student population. The general student 
population at HCC is nearly 50 percent white and roughly 25 percent each Hispanic and black, 
while the study sample is more evenly distributed among the three racial categories. Roughly 20 
percent of MAPS-eligible students were enrolling in college for the first time, approximately 
twice the percentage in HCC’s overall student population. 

Compared with the population of HCC students who were eligible for MAPS, the study 
sample looks quite similar. They match closely in gender and race. The study sample was 
slightly older and slightly more likely to be in their first semester in college, although these 
differences were quite small. 


Data Sources 

Several data sources are used in the analysis and presented in this report. 

• Baseline data: Baseline Infonnation Fonns (BIFs) were completed by all study 
participants before they were randomly assigned. The questionnaire asked 


3 Data for the study sample come from the BIF. HCC provided the information on the general HCC popu- 
lation and MAPS-eligible population. 
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The Performance-Based Scholarship Demonstration 
Table 2.1 

Selected Characteristics of Sample Members at Baseline 


Hillsborough Community College 


Characteristic 

Full 

Sample 

Gender (%) 

Male 

33.7 

Female 

66.3 

Age (%) 

18-19 years old 

23.3 

20-23 years old 

23.6 

24-26 years old 

12.9 

27-30 years old 

11.5 

3 1 years and older 

28.6 

Average age (years) 

27.0 

Race/ethnicity a (%) 

Hispanic 

30.7 

White 

30.5 

Black 

33.1 

Asian or Pacific Islander 

1.2 

Other 

4.4 

Married (%) 

17.9 

Missing 

9.5 

Number of children (%) 

0 

56.1 

1 

17.4 

2 

12.5 

3 or more 

14.0 

Household receiving any government benefits b (%) 

38.0 

Missing 

12.6 

Financial dependence 11 

Independent 

60.7 

Dependent 

34.2 

Missing 

5.1 

Currently employed (%) 

50.8 

Among those currently employed: 

Number of hours worked per week in current job (%) 

1-10 hours 

4.0 

11-20 hours 

24.0 

21-30 hours 

21.5 

31-40 hours 

47.1 

More than 40 hours 

3.4 


(continued) 
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Table 2.1 (continued) 


Characteristic 

Full 

Sample 

Highest grade completed (%) 

10 th grade or lower 

13.9 

1 1 th grade 

8.2 

12 th grade 

77.9 

Diplomas/degrees eamed d (%) 

High school diploma 

75.4 

GED certificate 

26.3 

Occupational/technical certificate 

13.7 

Associate's degree or higher 

1.3 

None of the above 

0.1 

Date of high school graduation/GED receipt (%) 

During the past year 

18.5 

Between 1 and 5 years ago 

31.9 

Between 5 and 10 years ago 

18.9 

More than 10 years ago 

30.7 

First semester at any college or university (%) 

23.7 

Expected enrollment in coming semester (%) 

Full time (12 credits or more) 

69.6 

Part time (6 to 1 1 credits) 

28.3 

Less than part time (less than 6 credits) 

2.1 

Main reason for enrolling in college 11 (%) 

To complete a certificate program 

1.9 

To obtain an associate's degree 

49.0 

To transfer to a 4-year college/university 

47.5 

To obtain/update job skills 

2.8 

Other 

0.9 

First person in family to attend college (%) 

33.3 

Language other than English spoken regularly in home (%) 

25.6 

Sample size 

1,075 


SOURCE: MDRC calculations from Baseline Information Form data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Characteristics shown in italics are calculated for a proportion of the full sample. 

Missing values are only included in variable distributions for characteristics with more than 5 percent of the 
sample missing. 

Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are included in the Other category. 
These respondents, combined with those who said they were American Indian or Alaska Native, or another 
race/ethnicity, are included in Other. 

b Benefits include unemployment/dislocated worker benefits, Supplemental Security Income or disability, cash 
assistance or welfare, food stamps, and Section 8 or public housing. 

c Students were counted as financially independent if they met any one of the following criteria as of random 
assignment: they were age 24 years or older, they were married, or they had one child or more. 

■^Distributions may not add to 100 percent because categories are not mutually exclusive. 
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The Performance-Based Scholarship Demonstration 
Table 2.2 

Selected Characteristics of HCC's Student Population 
Compared with MAPS Target Population and MDRC Evaluation Sample: 
Fall 2010, Spring 2011, and Fall 2011 Semesters 


Hillsborough Community College 


Characteristic 

All HCC 
Students 3 

MAPS-Eligible 

Students b 

Evaluation 

Sample 

Gender (%) 

Male 

44.0 

37.3 

33.7 

Female 

56.0 

62.7 

66.3 

Age (%) 

Younger than 18 

3.7 

0.0 

0.0 

18-19 years old 

21.1 

28.2 

23.3 

20-23 years old 

28.8 

28.3 

23.6 

24-26 years old 

11.3 

12.5 

12.9 

27-30 years old 

10.4 

10.6 

11.5 

3 1 years and older 

24.7 

20.4 

28.6 

Average age (years) 

26.9 

25.7 

27.0 

Race/ethnicity (%) 

Hispanic 

25.6 

29.3 

30.7 

White 

47.1 

31.1 

30.5 

Black 

22.1 

36.2 

33.1 

Asian or Pacific Islander 

3.7 

1.5 

1.2 

Other 

1.5 

1.8 

4.4 

Diplomas/degrees earned (%)° 

High school diploma 

70.9 

73.8 

75.4 

GED certificate 

8.8 

20.6 

26.3 

Occupational/technical certificate 

2.9 

1.2 

13.7 

Associate's degree or higher 

5.3 

0.1 

1.3 

None of the above 

17.9 

5.6 

0.1 

First semester at any college or university (%) 

12.0 

21.1 

23.7 

Total student count d 

92,828 

4,207 

1,075 


SOURCES: Hillsborough Community College calculations from enrollment data and MDRC calculations from 
Baseline Information Form data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

Missing values are only included in variable distributions for characteristics with more than 5 percent of the 
evaluation sample missing. 

“Students who enrolled in multiple terms are counted more than once in the values shown. 
b Students were counted as MAPS-eligible if they met the criteria described in Chapter 2. Data on age and 
Expected Family Contribution were not available for some students; these students were not counted as eligible. 
Students who enrolled in multiple terms are counted more than once in the values shown. 

“Distributions may not add to 100 percent because categories are not mutually exclusive. 
d Total student count for all HCC students and MAPS-eligible students includes some students more than once 
if they enrolled in multiple terms at HCC. 


19 







students to provide demographic and other background information. These da- 
ta are used to describe the sample, to assess the similarity of the research 
groups, and to define subgroups of sample members for analyses. 

• Field research: MDRC staff conducted three rounds of on-site field research 
that included interviews with staff, student focus groups, and Math Lab ob- 
servations (spring 2011, fall 2011, and spring 2012). Staff who were inter- 
viewed included the MAPS program coordinator, program administrative as- 
sistant, MAPS payment processor, Math Lab directors, the dean of math, the 
academic support coordinator, and the vice president of academic affairs. 
The student focus groups were conducted with program and control group 
students separately. The attendance in these focus groups ranged from three 
to thirteen students. The team conducted Math Lab observations to better un- 
derstand the program service environment. Five Math Lab observations were 
conducted at Brandon and six were conducted at Dale Mabry. Additionally, 
two tutor focus groups were conducted, one on each campus, during the last 
round of field research; the attendance in these focus groups ranged from 
seven to nine tutors. 

• Financial aid data: HCC provided MDRC with annual data on the financial 
aid amounts awarded to and received by students for each academic year that 
the program operated. 

• Integrated Postsecondary Education Data System (IPEDS) data: MDRC cal- 
culated the average cost per credit at HCC based on financial, enrollment, 
and instructional activity data reported to the U.S. Department of Education, 
National Center for Education Statistics, available online at IPEDS. 4 

• MAPS expenditure data: MDRC obtained cost infonnation from program 
expenditure and scholarship receipt data as reported by HCC to the Helios 
Education Foundation from July 2010 to January 2013. 

• Scholarship payment data: MDRC obtained data from HCC on scholarship 
payment records for all program group members for the program period. 
These data were used to describe rates of scholarship receipt. 

• Student services utilization data: HCC used the Who’s Next computerized 
tracking system to capture students’ time spent in the Math Labs and other 


4 See http ://nces. ed.gov/ipeds/datacenter. 
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student services centers. All students using the Math Lab centers were re- 
quired to sign in and sign out. Who ’s Next data were analyzed for all service 
centers that offered tutoring, including the Math Labs, for the program peri- 
od. ( Who ’s Next is discussed in more detail in Chapter 3.) 

• Student survey: MDRC administered a student survey to all three cohorts of 
sample members in spring 2012. The survey covered topics including sample 
members’ educational experiences (including visits to Math Labs), use of 
MAPS scholarship funds, motivation, and other topics. 

• Student transcript data: HCC provided MDRC with transcript data for sam- 
ple members. These data comprise various course outcomes including cours- 
es registered for, withdrawn from, and passed; number of credits earned; and 
the grade received in each course. This report includes two years of follow- 
up for each cohort of the MAPS sample. 
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Chapter 3 


Implementation, Student Participation, 
and Treatment Contrast 


Chapter 1 described the core components of the Mathematics Access Perfonnance Scholarship 
(MAPS) program model as it was designed. This chapter discusses how the program actually 
operated at Hillsborough Community College (HCC) — that is, how it was implemented in 
practice. In addition, this chapter explores how well the observed implementation of the 
program followed the intended design (sometimes referred to as “fidelity”), 1 and to what extent 
the experience of students in the program and control groups differed (sometimes referred to as 
“treatment contrast”). 2 


Main Findings 

The main findings are summarized below. 

• MAPS operated largely as designed. 

Program staff fulfilled their duties as expected, and scholarship payments and book 
bonuses were distributed with few errors. 

• More than 90 percent of students in the program group earned a MAPS 
award in the first semester, but rates of scholarship receipt declined to 
slightly less than 60 percent in the second semester and around 30 per- 
cent in the third semester. 

Rates of textbook receipt also dropped from around 36 percent in the first semester to 7 
percent in the third semester. These rates decreased partly as a result of fewer students reenrol- 
ling each semester: only around 80 percent and 60 percent of program group students reenrolled 
in the second and third semesters, respectively. However, even after accounting for students who 
did not reenroll at HCC, rates of scholarship receipt declined: more than 95 percent of program 
group students who were enrolled in the first semester received a MAPS payment, while only 
slightly more than 50 percent of students who were enrolled in the third semester did. 


'Mowbray, Ho Iter, Teague, and Bybee (2003). 
2 Weiss, Bloom, and Brock (2014). 
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MAPS students were much more likely to visit the Brandon and Dale 
Mabry Math Labs than control group students. 


MAPS students were 38 percentage points more likely than control group students to visit 
the Math Labs in the first semester — 87 percent of program group students visited the Math 
Labs, compared with 49 percent of control group students. A similar difference occurred in the 
second semester. In both the first and second semesters, MAPS students also spent more time and 
made more visits to the Math Labs on average than control group students — more than doubling 
the average number of visits and average number of hours spent there by control group students. 


Implementation of MAPS 

This section discusses everything that went into making MAPS operational at HCC — that is, 
all the staffing and activities that took place that led to students being able to experience MAPS. 
These details may be important for other institutions interested in setting up and operating 
MAPS or a similar program. 

Administrative Support 

MAPS was created in part because administrators strongly supported a developmental 
math intervention. Staff and administrators reported a keen understanding of the developmental 
math challenge faced by many HCC students and a basic belief in the MAPS theory of change 
to improve math outcomes for most developmental math students. Administrators further 
demonstrated their support of the program by allocating additional resources to increase tutor 
staffing at the Math Labs. The Vice President of Academic Affairs noted, 

A lot of students simply have high levels of math anxiety. . . [and] lack founda- 
tional skills. . ..The sheer volume of material that they have to cover in a single 
semester is a challenge. ... [I hope that MAPS] will add to their motivation. 

The support of administrators was key to the successful implementation of MAPS. 

Key Staffing Roles and Duties 

In addition to the administrators, many other HCC staff made significant contributions 
to MAPS. In order to support the program and study, the college designated staff members to 
manage and implement MAPS. The Vice President of Academic Affairs served as senior 
adviser to the project and assigned a full-time program coordinator to manage the day-to-day 
responsibilities of the program. The program was also supported by a number of staff across 
several departments and offices in the college, including financial aid, financial services, the 
office of institutional research (IR), and the Math Labs at Dale Mabry and Brandon. 
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Through most of the study, MAPS was staffed by one coordinator and supported by an 
administrative assistant. Their primary duties included recruiting students, planning and 
facilitating intake sessions, assisting students with registration for math courses, responding to 
program-related inquiries from students about scholarship payments and program requirements, 
making sure bonus textbooks and vouchers were distributed to students in a timely manner, and 
sending students program communications, such as e-mails notifying students of scholarship 
receipt and reminding students about program requirements. 

Enrollment and Grade Verification 

The process of verifying enrollment and grade data to determine which students earned 
the scholarship was conducted by staff from the office of institutional research. Throughout the 
program, a designated research analyst in the IR office kept track of which MAPS students 
remained enrolled in their developmental math classes beyond the add/drop period. These lists 
of students were then shared with staff in the financial aid office who issued the initial $100 
scholarship payments. 

At the end of each semester, a similar process was used to verify which students met the 
“C” or higher grade benchmark required to receive the final $500 scholarship. The IR analyst 
also noted which students met the “B” or higher grade benchmark and earned the bonus award. 
That information was shared with the program coordinator, who was responsible for purchasing 
and distributing the bonus awards to eligible students. 3 

Math Lab Visit Tracking 

In addition to completing a math course with a “C” or better grade, MAPS students 
were required to visit the Brandon or Dale Mabry Math Labs a minimum number of times and 
spend a minimum number of hours at the labs each semester in order to earn the final $500 
scholarship payments. 4 Tracking this component required cooperation among the program staff, 
the IR analyst, and the staff who managed the Math Labs. 

The college used Who ’s Next, a software program for data content management and 
tracking, to determine the number of visits students were making and how much time they were 
spending in the Math Labs. This software required all students at both campuses to sign in when 
they entered the lab, choose their activities (including tutoring, computer use, and paper-and- 
pencil work), and sign out when they left the lab. 


3 In some instances book vouchers that covered the cost of textbooks were distributed to students instead of 
actual textbooks. 

4 See Chapter 1 for more information about the MAPS program model and Math Lab requirements. 
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During the first semester, the sign-in and sign-out processes were not uniform across 
the Dale Mabry and Brandon campus Math Labs, and there were questions about the accuracy 
of the Who 's Next data. A few students asked the program coordinator whether their hours were 
correct. The Math Lab managers, working with the program coordinator, found that there were 
some inaccuracies in the data; together, they reviewed and corrected the data. In subsequent 
semesters, Math Lab staff were trained to make sure students signed in and out uniformly, 
ensuring that students’ logged visits and hours would be accurate. 

Partly because of these issues, by the second semester the responsibility of reviewing 
the lab visit data was transferred to the IR analyst, who was given access to retrieve this infor- 
mation directly from the Who ’s Next database. This decision was also made to streamline the 
scholarship verification process. The analyst verified which students met the grade benchmark 
as well as the Math Lab visit requirements and sent that list to financial aid. 

Other enhancements were made to the Who ’s Next tracking system over time. In partic- 
ular, one addition to the software enabled students to view time spent in the lab over the course 
of a semester. When students signed in to the lab through the system, a text box appeared on the 
screen indicating how many hours they had spent in the lab since the start of the semester. 
While this appeared for all students using the lab, it enabled program group students to keep 
track of how close or far they were from meeting the Math Lab requirements. Additionally, this 
development decreased the number of student inquiries to Math Lab staff or the coordinator 
about their hours. In interviews with MDRC staff, students reported that they appreciated this 
update to Who ’s Next. 

As a result of these changes, the tracking of service requirements improved over time. 
While most students who participated in focus groups during the spring 2011 semester agreed 
that the Math Lab tracking system was “susceptible to glitches,” in fall 2011 most students 
reported that “the hours were accurate.” 

Financial Aid 

One of the eligibility criteria for enrolling in the study was that students had to have an 
Expected Family Contribution (EFC) of 5,273 or below, the threshold for Pell Grant eligibility 
at that time. MDRC worked with HCC financial aid staff to ensure that the MAPS scholarship 
payments did not have a negative impact on students’ financial aid packages, as the scholarship 
was intended to be awarded on top of existing aid. Financial aid officers were well acquainted 
with the fact that students’ financial aid packages cannot exceed the total cost of attendance. 5 If 

5 A student’s unmet financial need is calculated as the cost of attendance (including tuition, fees, books, 
and living expenses) minus the student’s financial aid package minus the student’s EFC. 
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students are offered aid above the total cost of attendance, then additional aid — in this case, the 
scholarship — cannot be added until another form of aid is replaced. 6 To ensure that students 
were aware of the potential impact on their aid packages, financial aid officers reviewed 
packages of all students who registered to attend a MAPS information session. Those students 
for whom adding the scholarship (if they were assigned to the program group) would require a 
replacement of aid were informed individually before they agreed to participate in the study. 

Scholarship Payment Administration 

Once the IR analyst determined which students met the Math Lab visit and grade re- 
quirements, the department sent the lists of students who achieved the benchmarks to the 
financial aid office. During the first semester of MAPS, a small number of incorrect payments 
were made to students. 7 By the second semester, the college designated someone from the 
college’s financial services office, which deals with payment distribution, to take on the role of 
payment processor for the MAPS program in order to make the process more seamless. Having 
one person with a background in financial aid and financial services designated to handle the 
MAPS payments helped to prevent future errors. This payment processor received the lists of 
students who met the benchmarks from the IR analyst and ensured that the correct students 
received their scholarship payments. 

HCC Math Lab Services 

HCC’s Dale Mabry and Brandon Math Labs are learning assistance centers that offer a 
variety of supports to students. During the study, the labs were staffed with tutors who were 
available to work with students in one-on-one appointments or in groups. Additionally, the labs 
were each equipped with approximately 40 computers that had the MyMathLab and Math XL 
software installed. Both MyMathLab and Math XL were created by the same textbook manu- 
facturer. MyMathLab is designed to provide students with online practice, course materials, and 
video lectures, while Math XL is “an online homework, tutorial and assessment system.” 8 The 
Math Labs also offer hard-copy worksheets for each developmental math course, corresponding 


6 Typically, the scholarship would replace aid in the form of a loan. If aid for a MAPS program student 
was repackaged and the student later decided to readjust the package and add the loan back, a change could be 
requested through the Financial Aid office at any point before the end of the semester. 

7 In the first semester, four erroneous payments were made to students who did not meet the scholarship 
criteria. Additionally, one student who met the final payment criteria did not receive the award. HCC was 
notified and followed up with the student. One additional erroneous payment was found in the second 
semester. 

8 See http://mxlmkt.pearsoncmg.com/product-overviews. 
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to specific chapters in students’ textbooks. Definitions, concepts, examples, and problems for 
students to solve are provided on the worksheets. 

The intention of the incentive to visit the Math Lab was to expose students to this aca- 
demic support service, encouraging a base level of usage with the intention that students would 
see the value of this service and use the centers as needed, beyond the minimum required to 
earn the scholarship payment. MAPS did not dictate to students how to spend their time in the 
Math Lab, allowing them, rather, to determine for themselves the best way to use the labs’ 
resources. All Math Lab services were available equally to control group students; they simply 
were not provided an incentive by MAPS to attend. 

MDRC conducted 1 1 observations of student usage of the Math Labs, 5 at Brandon and 
6 at Dale Mabry. 9 (Since the labs served all students, the activities of program or control 
students specifically may or may not have been observed.) Almost all students observed 
walking into the labs signed in at the front desk and selected their activities on a computer, 
which were recorded into the Who ’s Next database. Most students were observed signing out as 
well, which allowed the database to capture how much time each student spent at the Math Lab 
during each visit. During observations, students used each type of area designated for different 
types of math support activity, including areas with desktop computers set up individually for 
students to use Math XL and MyMathLab; areas in which students could sit next to their 
classmates to do group work or work with tutors in groups; and areas for students to work 
individually with tutors. Observations indicated a high level of engagement with all math 
activities in the labs. 

Overall, most students observed in the Math Labs did independent, computer-based 
work. About 25 percent of students were observed interacting with tutors. Typically, six to 
seven tutors staffed the labs during the observations. Students were observed raising their hands 
to ask tutors to assist them with both computer and paper-and-pencil work. Students also 
worked with tutors individually and with groups of other students. Tutors served all students, 
not just MAPS students specifically. 


9 Once Dale Mabry students complete Intermediate Algebra and begin college-level math, they must go to 
the “Learning Commons” if they want assistance from a tutor — they cannot continue to visit the Math Lab. 
MDRC conducted three observations of the Dale Mabry Learning Commons, which indicated that most 
students who visit sign in and go to the computers to work independently. Students seeking tutoring are 
required to make appointments in advance, typically by reviewing hard-copy schedules showing tutors’ future 
availability. Appointments are in high demand; students may need to wait a few days before spaces are 
available. Most appointments appeared to last 30 minutes. Tutors do not "float” (that is, move from one student 
to another) in the Learning Commons as they do in the Math Labs. 
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Tutoring 

During the program’s operation, there were about 15 tutors assigned to work at the 
Brandon campus and 26 tutors assigned to work at the Dale Mabry campus Math Lab each 
semester. Most of the tutors at the Math Labs were either HCC students or students at nearby 
University of South Florida — that is to say, a peer tutoring model was predominantly used. 
The Math Lab at Dale Mabry, the larger campus, also employed a few instructors and adjunct 
faculty members as tutors. 

Research on best practices in tutoring indicates that tutors should receive standardized 
training. 10 HCC chose the National Tutoring Association (NT A) to certify both the Dale Mabry 
and Brandon Math Lab tutoring programs and the individual tutors in each lab. 11 In order to 
qualify for certification through the NT A, staff had to receive 10 to 12 hours of training and 
professional development, divided into one- to two-hour modules made available over the 
semester. The content of the training includes both mathematics skills and how to tutor and 
work with students. 

Before the study began, tutoring certification training was required at Brandon’s Math 
Lab but not at Dale Mabry’s. In order to better standardize the quality of tutoring services across 
campuses, the college decided to require Dale Mabry Math Lab tutors to complete the same 
certification training. In May 2011, Math Lab tutors at Dale Mabry and Brandon received 
training for tutoring certification using the NTA program. Certification training was provided 
by the Math Lab directors, who ensured that training time did not interfere with academic 
services provided for students. Training was built into regular staff meetings and tutors were 
compensated for time spent in training and meetings. 

Two focus groups were held with tutors, one on each campus, during the final round of 
site visits. The tutors identified themselves as former HCC students, current students at the 
University of South Florida, and adjunct faculty. When asked about developmental math 
students’ primary challenges, some tutors noted “lack of math vocabulary,” or a basic deficien- 
cy in the fundamental mathematics concepts that make doing math and learning new math 
concepts easier. Tutors explained that these deficiencies usually lead to overdependence on 
tutors to walk students through their entire homework assignments. When discussing their own 
goals for students in the Math Labs, tutors expressed a desire for all students to internalize the 
lessons by working out the math problems on their own. These tutors emphasized that they 
usually help a student with one problem, then move on to the next student. The idea was not to 


10 Rutschow and Schneider (2011). 

"According to their website, “The NTA is now the oldest and largest association dedicated exclusively to 
tutoring.” See www.ntatutor.com/about.html. 
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Box 3.1 

How to Read the impact Tables in This Report 

Most tables in this report use the format illustrated in the abbreviated table below, which 
displays some survey data for the program and control groups that were part of the MAPS 
program at Hillsborough Community College. This shows that 94 percent of program group 
students visited the Math Lab, while 74.7 percent of control group students did so. 

The “Difference” column in the table shows the difference in outcome between the two 
research groups — that is, the estimated average impact of the opportunity to participate in the 
program. For example, the estimated average impact on Math Lab visits can be calculated by 
subtracting 74.7 from 94.0, yielding an impact estimate of 19.4 percentage points. This differ- 
ence represents the estimated average impact rather than the true average impact (which is 
impossible to determine) because, although study participants are randomly assigned to the 
program and control groups, the impact estimate would have been different if a different 
sample of students had been included in the study or if the same group of students had been 
randomized in a different way. 

Differences marked with one asterisk or more are considered statistically significant, meaning 
that there is a high probability that the opportunity to participate in the program had an impact 
on that outcome measure. Differences that have no asterisks indicate that the opportunity to 
participate in the program did not have a discernible effect on that outcome. The number of 
asterisks indicates the probability that an impact at least as large as the one observed in the 
study would have occurred even if the true average impact had been zero. One asterisk corre- 
sponds to a 10 percent probability; two asterisks, a 5 percent probability; and three asterisks, a 

(continued) 


spend a lot of time with each student who sought help, which in turn fosters students’ greater 
independence and confidence in their ability to solve math problems. Tutors reported that they 
most commonly helped students by “floating” in the labs — going from one student to the other 
and answering questions. During Math Lab observations, floating was the most common way 
tutors were seen interacting with students. 

Tutors at both campuses also discussed collaborative tutoring — that is, getting students 
to work with one another on shared problems — as their preferred approach to tutoring. As one 
tutor described the lab, “It’s very dynamic — sometimes there are three tutors and three groups 
of students working together!” Both Brandon and Dale Mabry tutors also spoke highly of the 
use of test corrections; in this scenario, instructors ask students to take the errors they made on 
exams to the Math Lab to correct them with tutor assistance. 


Program Participation and Treatment Contrast 

In order for MAPS to have academic impacts, the program needed to create a treatment 
contrast — that is, program group students needed to have different experiences from their 
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Box 3.1 (continued) 

1 percent probability. The more asterisks, the more likely the opportunity to participate in the 
program had a true average impact on the outcome. The impact in the table excerpt below has 
three asterisks, indicating that the impact is statistically significant at the 1 percent level — 
meaning that there is only a 1 percent chance of observing an estimated average impact this 
large (or larger) if the opportunity to participate in the program actually had no average effect 
on credits earned. In other words, there is a 99 percent level of confidence that the opportunity 
to participate in the program had a positive impact on the average number of credits earned. 

Also shown in the table is the standard error of the impact estimate. The standard error is a 
measure of uncertainty or variability around the impact estimate. Some useful rules of thumb 
are that there is about a 90 percent chance that the true average impact is within plus or minus 
1.65 standard errors of the estimated average impact, roughly a 95 percent chance that the true 
average impact is within plus or minus 1 .96 standard errors of the estimated average impact, 
and about a 99 percent chance that the true average impact is within plus or minus 2.58 stand- 
ard errors of the estimated average impact. 

The survey tables included in this report list sample size for each individual measure, as 
students who responded to the survey may not have answered every question. The impact 
tables show sample size in the last row, as the sample size for all measures is the same. 


Outcome 

Sample 

Size 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Ever visited an HOC Math Lab (%) 

846 

94.0 

74.7 

[Q 4 *** 

2.3 


control group counterparts. This section examines students’ program participation and 
the treatment contrast that ensued, including scholarship receipt, financial aid receipt, and the 
use of Math Lab services. Additionally, the section presents some survey results on differ- 
ences in educational experiences, employment, and motivation. Box 3.1 provides infonnation 
on how to read the impact tables in this and subsequent chapters. For more information about 
the survey, including information about the students who responded to the survey, see 
Appendix B. 

Time Periods Used in This Chapter 

The tables and figures presented in this chapter group HCC’s fall, spring, and summer 
semesters together in one of two ways, both of which are illustrated in Figure 3.1. Most of the 
tables and figures are organized into “first semester” through “fourth semester.” Here, the 
“first semester” refers to the semester in which students were randomly assigned into the 
study; the “second semester” refers to the next semester, and so on. For the fall 2010 cohort, 
for example, the “first semester” is the fall 2010 semester. In order to more effectively make 
comparisons between fall and spring cohorts, spring and summer semesters are combined in 


31 





The Performance-Based Scholarship Demonstration 
Figure 3.1 
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this organizational scheme. For the spring 201 1 cohort, therefore, the “first semester” includes 
both the spring 2011 and summer 2011 semesters. 12 Figure 3. 1 shows how these periods were 
constructed for each cohort. 13 

For financial aid data, only annual values were available. Financial aid results are there- 
fore presented in periods of first academic year and second academic year. Each academic year 
consists of a sequential set of fall, spring, and summer semesters. 14 The “first academic year” is 
the academic year in which students were randomly assigned into the study. Figure 3.1 summa- 
rizes these time periods for each cohort. 

Scholarship Receipt 

As shown in Table 3.1, rates of scholarship receipt were initially high: in the first se- 
mester, more than 92 percent of program group students received an initial payment for enrol- 
ling in a course in the MAPS sequence. Nearly 54 percent of the program group met the 
relevant Math Lab requirements and passed their course, receiving a final payment, while 36 
percent of program students earned a grade of “A” or “B” and received a textbook in the 
following semester. 

The rate of scholarship receipt fell in the second semester: only 58 percent of program 
students received a MAPS payment, with 53 percent receiving an initial payment for enrolling 
in a course in the MAPS sequence and less than 37 percent receiving a final payment for 
passing a MAPS course and meeting associated Math Lab requirements. 15 A smaller proportion 
of the program group, 16.5 percent, received a textbook in the third semester for earning a grade 
of “A” or “B” in the second semester. However, fewer than 80 percent of study students 
registered at HCC in the second semester, so a substantial portion of the drop in scholarship 
receipt was driven by students who did not enroll at HCC at all. 


12 Rates of enrollment in summer semesters were low compared with enrollment rates in the spring and fall 
semesters. As a result, combining values for spring and summer semesters into a single semester (for example, 
“first semester”) yields values that are generally similar to the spring semester alone. 

13 Different time periods are used in the analyses presented in Chapters 3 and 4. In Chapter 3, spring and 
summer semesters are always combined when counting semesters following random assignment. In Chapter 4, 
spring and summer semesters are treated separately for the spring 2011 cohort in order to calculate more 
comparable estimates of program effects across all cohorts. Figures 3.1 and 4.1 illustrate the differences 
between the time periods used in the two chapters. 

14 Data for summer 2013 were not available as of the writing of this report. 

15 In the second and third semesters, some students received either only an initial payment or a final pay- 
ment, but not both. As a result, the proportion of students who received any payment in those semesters was 
higher than the proportion who received either type of payment individually. 
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The Performance-Based Scholarship Demonstration 
Table 3.1 

Scholarship Receipt Among Program Group Members 


Hillsborough Community College 


Outcome 

Program 

Group 

First semester 

Registered for any course (%) 

95.7 

Received one or more scholarship payments (%) 

92.6 

Received initial payment 

92.6 

Received final payment 

53.9 

Received a textbook 3 

36.1 

Average scholarship amount b ($) 

398 

Average scholarship amount among recipients 

430 

Second semester 

Registered for any course (%) 

79.5 

Received one or more scholarship payments (%) 

58.2 

Received initial payment 

52.7 

Received final payment 

36.8 

Received a textbook 3 

16.5 

Average scholarship amount b ($) 

272.7 

Average scholarship amount among recipients 

473.7 

Third semester 

Registered for any course (%) 

59.8 

Received one or more scholarship payments (%) 

30.3 

Received initial payment 

25.7 

Received final payment 

18.7 

Received a textbook 3 

7.0 

Average scholarship amount b ($) 

119.1 

Average scholarship amount among recipients 

393.6 

Sample size 

674 


SOURCES: MDRC calculations from Hillsborough Community College scholarship payment data and 
transcript data. 

NOTES: Characteristics shown in italics are calculated for a subset of the program group. 

Semesters consist of either a fall semester only or a spring and summer semester combined. As a result, 
some students may earn up to two initial or final scholarship payments in a single program semester (once in 
spring, and again in summer). These students are counted only once when calculating the percentage of students 
who received payments. However, their scholarship amount reflects all payments that they received in the 
spring and summer semesters. 

a Values shown above reflect only the students who redeemed their vouchers the following semester, not all 
students who were given a voucher; the latter value is not available. 
b Scholarship amount includes initial and final awards and excludes textbook price. 
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Rates of scholarship receipt declined again in the third semester. Slightly more than 30 
percent of program group students received a MAPS payment of any type. Almost 26 percent of 
the program group received an initial payment for registering for a course in the MAPS se- 
quence, and nearly 19 percent received a final payment for passing the class and meeting 
relevant Math Lab requirements. Only 7 percent of program group students received a textbook 
in the fourth semester for earning a grade of “A” or “B” in their MAPS course in the third 
semester. As in the second semester, the decreasing rate of scholarship receipt was driven in 
part by program group students who did not enroll at HCC: less than 60 percent of students 
enrolled at HCC in the third semester. 

The declining portion of students enrolled at HCC in each successive semester was an 
important factor in the drop in scholarship receipt. Figure 3.2 illustrates that this decline in 
enrollment was comparable for both program and control group students. While more than 90 
percent of students enrolled in the first semester, fewer than 80 percent of either the program or 
control group enrolled in the second semester. Tins downward pattern continued in the third and 
fourth semesters, with only approximately 60 percent and 50 percent of both groups enrolling in 
each semester, respectively. However, even after accounting for students who did not reenroll at 
HCC, rates of scholarship receipt declined: more than 95 percent of program group students who 
were enrolled in the first semester received a MAPS payment, while only slightly more than 50 
percent of students who were enrolled in the third semester did. Lower rates of scholarship 
receipt in the second and third semesters may therefore also have been driven by lower rates of 
success among program group students who were still enrolled at HCC, as these students either 
failed courses in the MAPS sequence or chose not to enroll in these courses at all. 

Student Opinions and Use of Scholarship Dollars 

Program group students were surveyed regarding their opinions of and primary use of 
the MAPS scholarship. 16 As shown in Table 3.2, more than 80 percent of program group 
students who could recall participating in the program considered the scholarship either “useful” 
or “very useful” for academic success, and more than 75 percent of the students considered 
it either “important” or “very important” financially for staying in school. 17 Furthennore, many 


16 The student survey covered a variety of topics, including students’ educational experiences, experiences 
with the Math Labs, experiences with the MAPS program, self-confidence, motivation, work experiences, 
financial situation, and psychological well-being. The survey response rate was 79 percent. Two analyses for 
potential biases in survey responses were conducted; the analyses found that students who responded to the 
survey may differ from those who did not with regard to certain baseline characteristics. See Appendix B for 
further discussion. 

17 Although students considered the scholarship important for their ability to stay in school financially, the 
scholarship program did not have an impact on students’ persistence — see Chapter 4. 
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The Performance-Based Scholarship Demonstration 
Figure 3.2 

Student Enrollment During the First Through Fourth Semesters 
Hillsborough Community College 
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SOURCE: MDRC calculations from Flillsborough Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre -random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


students in focus groups stated that the quality of their math instruction, more so than either 
MAPS scholarship dollars or tutoring, had the greatest influence on their success in math. These 
findings serve as reminders that while many students found the scholarship dollars useful, other 
factors also played substantial roles in students’ persistence and academic success. 

More than 90 percent of program group students who could recall receiving a MAPS 
payment reported that they used the MAPS awards primarily either to pay for school (42.5 
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The Performance-Based Scholarship Demonstration 
Table 3.2 

Student Opinions and Use of Scholarship Dollars Among Program Group Members 

Hillsborough Community College 


Outcome (%) 

Sample 

Size 

Program 

Group 

Recall being a participant in MAPS 

549 

99.3 

Of those who recall being a participant in MAPS 

Recall receiving e-mails from MAPS 

544 

98.7 

Knew minimum grade needed for performance payment 

544 

86.8 

Remember receiving payment from MAPS 

530 

88.5 

MAPS useful/very useful to academic success 

545 

80.2 

MAPS important/very important for staying in school financially 

544 

75.9 

Of those who recall receiving a MAPS payment 
Main use of scholarship 

To pay for school (tuition, fees, books, etc.) 

468 

42.5 

To help with basic living expenses (bills, food, housing, etc.) 

468 

47.9 

To help with child care costs 

468 

1.9 

To buy clothes or shoes 

468 

0.2 

To deposit into bank account 

468 

3.4 

To buy something normally could not afford or entertainment 

468 

1.7 

To work fewer hours at job 

468 

1.3 

Some other way 

468 

0.2 

Don't remember 

468 

0.9 

Sample size 

549 



SOURCE: MDRC calculations using responses from the Performance-Based Scholarship Hillsborough Community 
College survey. 

NOTES: Characteristics shown in italics are calculated for a subset of the program group. 

Missing values are not included in individual variable distributions. 


percent) or to help with basic living expenses such as bills, food, and housing (47.9 percent.) Far 
fewer program group students reported using the award primarily for other things, such as child- 
care costs, clothes, or savings. 

Financial Assistance 

As described previously, the MAPS scholarship was intended to be awarded on top of 
all other financial aid and not to displace other forms of aid. An analysis of financial aid data, 
shown in Table 3.3, illustrates that the program functioned as intended, and that there was a 
difference for program and control group students: in their first academic year of MAPS, the 
number of program group students receiving aid was 3.6 percentage points greater than the 
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The Performance-Based Scholarship Demonstration 
Table 3.3 

Impacts on Annual Financial Assistance During the First Two Academic Years 


Hillsborough Community College 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First academic vear 

Received any financial assistance (%) 

97.8 

94.2 

3.6 *** 

1.2 

Received Pell Grant 

92.9 

93.0 

-0.1 

1.6 

Received MAPS award 3 

95.0 

0.2 

94 7 *** 

1.1 

Received other grants 6 

26.5 

25.0 

1.5 

2.8 

Received subsidized loans 

29.9 

29.4 

0.5 

2.9 

Received unsubsidized loans 

22.1 

19.5 

2.6 

2.6 

Received private loans 

0.0 

0.4 

-0.4 

0.3 

Received work-study 

2.2 

1.0 

1.3 

0.8 

Average annual financial assistance received ($) 

7,378 

6,214 

1,163 *** 

261.7 

Pell Grant 

4,131 

4,029 

103 

119.0 

MAPS award 3 

767 

2 

765 *** 

27.6 

Other grants 6 

455 

353 

101 * 

60.2 

Subsidized loans 

1,024 

1,013 

11 

101.3 

Unsubsidized loans 

916 

764 

152 

117.4 

Private loans 

2 

19 

-17 

11.9 

Work- study 

82 

34 

48 

35.8 


Percentage of financial assistance received by recipients (%) 


As grants/scholarships 

82.9 

81.8 



As loans 

16.5 

18.0 



As work-study 

0.6 

0.2 



Second academic vear 

Received any financial assistance (%) 

64.6 

61.0 

3.7 

3.1 

Received Pell Grant 

58.4 

57.5 

0.9 

3.1 

Received MAPS award 3 

38.2 

0.0 

3g 9 *** 

2.4 

Received other grants 6 

22.7 

21.9 

0.8 

2.6 

Received subsidized loans 

37.2 

37.1 

0.1 

2.9 

Received unsubsidized loans 

32.9 

32.4 

0.5 

2.7 

Received private loans 

0.0 

0.0 

0.0 

0.0 

Received work-study 

2.2 

1.1 

1.1 

0.8 

Average annual financial assistance received ($) 

5,602 

5,035 

568 * 

344.5 

Pell Grant 

2,144 

2,086 

58 

137.7 

MAPS award 3 

195 

0 

295 *** 

15.9 

Other grants 6 

346 

287 

59 

48.3 

Subsidized loans 

1,397 

1,347 

51 

110.5 

Unsubsidized loans 

1,429 

1,300 

129 

129.2 

Private loans 

0 

0 

0 

0.0 

Work- study 

92 

19 

73 ** 

34.3 


(continued) 
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Table 3.3 (continued) 


Program 

Control 

Standard 

Outcome 

Group 

Group Difference 

Error 

Percentage of financial assistance received by recipients (%) 




As grants/scholarships 

61.1 

59.0 


As loans 

37.8 

40.7 


As work-study 

1.2 

0.3 


Sample size (total = 1,075) 

674 

401 



SOURCE: MDRC calculations from Hillsborough Community College financial aid data. 


NOTES: Roimding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre-random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between the research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; and * = 10 percent. 

Outcomes shown in italics are nonexperimental. Statistical significance tests are not conducted on 
nonexperimental outcomes. 

The academic year is defined as fall through summer. 

For the fall 2010 and fall 201 1 cohorts, students were eligible to earn MAPS awards in all three semesters 
included in the first academic year. For the spring 201 1 cohort, students were only eligible to earn MAPS awards 
during the spring and summer semesters of the first year. The second year of data includes one fall semester in 
which students in the fall 2010 and fall 201 1 cohorts could earn MAPS awards, and two semesters (fall and 
spring) in which students in the spring 2011 cohort could earn MAPS awards. 

a In general, only program group students received MAPS awards. One control group student received a 
MAPS award erroneously. 

b This includes all grants and scholarships except for Pell Grants and MAPS awards. 


number of control group students who received aid. This result was surprising, as the program 
was designed to increase the amount of aid that students receive, but not necessarily to extend 
aid to a greater proportion of students. Program group students also received an average of 
$1,163 more in aid than did control group students. 18 These differences in average aid received 
were mainly due to MAPS awards: program group members received an average of $767 in 
MAPS awards, while control group members received close to $0. 19 The difference in average 
total amount of aid received suggests that the program was working as intended, and that MAPS 
awards were truly being treated as “last dollar” awards. 


18 These values reflect the average aid amounts for all cohorts during their first academic year of MAPS, 
beginning with the fall semester and ending with the following summer semester. Data were not available for 
individual semesters broken out separately. Because the data were only available for each academic year, the 
first academic year includes one fewer semester of MAPS aid for students who enrolled in the program 
beginning in a spring semester, while the second academic year includes fewer semesters of MAPS aid for 
students who enrolled in the program beginning in a fall semester. 

l9 One control group member was accidentally awarded a MAPS scholarship. 
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Similar differences were also found for the second academic year of the program, alt- 
hough they were smaller. In the second academic year, as in the first, there is no evidence that 
the MAPS scholarship displaced other awards from students’ financial aid packages. 

Math Lab Use 

As noted in Chapter 1, HCC faculty and administrators were concerned that students 
were underutilizing Math Lab services and thought that increased use of the Math Labs would 
help drive better academic outcomes. MAPS was therefore designed to encourage greater 
student use of the Brandon and Dale Mabry Math Labs. A review of Who ’s Next tracking data 
for the labs confinns that program group students made greater use of the labs than did control 
group students. Table 3.4 shows the levels of use of the Math Labs by program and control 
group students and the differences between the two groups. Throughout this period, program 
group members consistently used the Brandon and Dale Mabry Math Labs at higher levels than 
did control group students. Program group members were more likely to visit the Math Labs, 
had a higher average number of visits, and spent more time there on average. The program also 
appeared to significantly increase the proportion of students who used Math Lab services 
heavily (5 hours or more per semester or 10 hours or more per semester). For example, in the 
first semester, 35.6 percent of program group students spent 10 hours or more at the Math Labs, 
compared with only 15.0 percent of control group students. 20 

In focus groups, program group students generally had positive things to say about the 
Math Labs and the tutors. They reported that the workshops and end-of-semester reviews were 
helpful, and that most tutors were knowledgeable. As one student said, “With the Math Lab, 
there [are] tutors and floaters, and you [can] get your question answered in a maximum of 10 to 
15 minutes. Then you [can] understand it, do more, and when you have another question, 
someone [is] there.” One student who had finished her developmental math courses and moved 
on to college-level math said, “For this class, I am not required to take any tutoring for MAPS, 
but I’m still there [at the Math Lab] every day to do homework or ask questions.” Some students 
also said that studying in the lab helped them stay motivated and on task. Although several 
students reported that the Math Labs can get noisy, students agreed that the lab was a good place 
to do work without too many distractions, and that visiting the Math Lab was easy to fit into their 
schedules before or after class, since tutoring appointments were not required. One student made 
one-hour appointments with various tutors and said, “They would definitely come help me if I 
had a problem.” She reported spending about 70 hours in the Math Lab each semester, far 


20 Students enrolled in Beginning Algebra were required to spend five hours in the Math Lab in order to 
receive their final $500 MAPS payment. Later MAPS courses had lower Math Lab requirements. 
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The Performance-Based Scholarship Demonstration 
Table 3.4 

Learning Assistance Center Use 


Hillsborough Community College 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First semester 

Registered for any course (%) 

95.8 

93.9 

1.9 

1.4 

Used any learning assistance center (%) 

88.5 

60.4 

28.1 *** 

2.5 

Brandon or Dale Mabry Math Labs 

86.8 

48.9 

37 g *** 

2.6 

Other learning assistance centers 

31.1 

28.5 

2.7 

2.9 

Number of visits to any learning assistance center 

11.5 

4.9 

6.6 *** 

0.8 

Brandon and Dale Mabry Math Labs 

9.7 

3.5 

^ i *** 

0.6 

Other learning assistance centers 

1.9 

1.4 

0.5 

0.4 

Hours spent at any learning assistance center 

14.5 

5.7 

g g *** 

1.5 

Brandon or Dale Mabry Math Labs 

12.4 

4.5 

7 g *** 

1.2 

Other learning assistance centers 

2.1 

1.2 

0.9 

0.6 

Hours spent at Brandon or Dale Mabry Math Labs (%) 

5 hours or more 

69.6 

22.9 

46.7 *** 

2.9 

10 hours or more 

35.6 

15.0 

20.6 *** 

2.8 

Second semester 

Registered for any course (%) 

79.5 

78.1 

1.3 

2.6 

Used any learning assistance center (%) 

62.2 

36.6 

25.6 *** 

3.1 

Brandon or Dale Mabry Math Labs 

56.0 

23.2 

32 8 *** 

3.0 

Other learning assistance centers 

24.1 

21.7 

2.4 

2.7 

Number of visits to any learning assistance center 

5.8 

2.8 

3 q *** 

0.6 

Brandon and Dale Mabry Math Labs 

4.4 

1.6 

2 g *** 

0.4 

Other learning assistance centers 

1.5 

1.2 

0.3 

0.3 

Hours spent at any learning assistance center 

6.9 

3.1 

3 g *** 

0.8 

Brandon or Dale Mabry Math Labs 

5.4 

2.0 

3 4 *** 

0.6 

Other learning assistance centers 

1.5 

1.1 

0.4 

0.4 

Hours spent at Brandon or Dale Mabry Math Labs (%) 

5 hours or more 

29.2 

10.3 

jg 9 *** 

2.6 

10 hours or more 

15.6 

6.4 

g 2 *** 

2.1 


(continued) 


more than the hours required for MAPS. While some focus group students said that they knew 
about the Math Lab before joining MAPS, one student credited MAPS with her math success: 

I’ll be honest, 1 probably wouldn’t have known about the Math Lab if not for 
MAPS. 1 probably would have taken the courses online instead. Before this pro- 
gram, 1 took like three years off, and I tided an online course and I was on my 
own without the Math Lab. 


41 






Table 3.4 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Third semester 

Registered for any course (%) 

60.2 

57.9 

2.4 

3.1 

Used any learning assistance center (%) 

31.7 

23.5 

g 2 *** 

2.9 

Brandon or Dale Mabry Math Labs 

22.2 

11.3 

10.9 *** 

2.4 

Other learning assistance centers 

16.5 

16.9 

-0.4 

2.4 

Number of visits to any learning assistance center 

2.8 

1.6 

11** 

0.5 

Brandon and Dale Mabry Math Labs 

1.6 

0.6 

1 o *** 

0.3 

Other learning assistance centers 

1.2 

1.0 

0.2 

0.3 

Hours spent at any learning assistance center 

3.9 

2.0 

1.9 ** 

0.8 

Brandon or Dale Mabry Math Labs 

2.4 

1.0 

1 .4 ** 

0.7 

Other learning assistance centers 

1.5 

1.0 

0.4 

0.5 

Hours spent at Brandon or Dale Mabry Math Labs (%) 

5 hours or more 

10.1 

4.2 

5 g *** 

1.7 

10 hours or more 

5.9 

2.1 

2 7 *** 

1.3 

Sample size (total = 1,075) 

674 

401 




SOURCE: MDRC calculations from Hillsborough Community College Who’s Next data on tutoring service use. 

NOTES: Rounding may cause slight differences in sums and differences. Estimates are adjusted by cohort-campus 
interaction, National Center for Education Statistics risk factors, and selected pre-random assignment placement 
test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The tutoring requirements for the MAPS scholarship are a minimum of 5 hours and 5 visits to the Math Lab for 
those enrolled in Beginning Algebra and a minimum of 3 hours and 3 visits to the Math Lab for those enrolled in 
Intermediate Algebra. There is no tutoring requirement for those enrolled in college-level math courses. 

Learning assistance centers used in this analysis include the Brandon and Dale Mabry Math Labs and other 
centers across all campuses that offered tutoring services. 

Semesters consist of either a fall semester only, or spring and summer semester combined. 


Program and control group members’ rates of utilization were also compared at other 
learning assistance centers that offered tutoring services, including centers on other campuses, 
to determine whether the MAPS scholarship was causing students to use these centers less in 
order to spend more time at the Brandon and Dale Mabry Math Labs. Analysis of Who 's Next 
data showed that both program and control groups used other student services centers at 
comparable rates. Both groups of students had comparable average numbers of visits to the 
other centers and spent comparable amounts of time at them. This suggests that the additional 
time spent at the MAPS Math Labs did not cause students to spend less time at other tutoring 
locations. 
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Educational Experiences and Other Mediators 

In addition to analyzing HCC Who ’s Next records to estimate the contrast in services 
received, MDRC surveyed both program and control group students regarding a variety of 
topics in order to analyze whether the program changed students’ experience at HCC in other 
ways. The survey included academic topics such as students’ study habits, their experience at 
the Math Labs, and their use of tutors, as well as nonacademic topics such as students’ em- 
ployment and reasons for wanting to do well in school. All three cohorts of students were 
surveyed in the spring 2012 semester, meaning that different cohorts were at different points in 
the math sequence when they were surveyed. 21 Survey results showed that the program had 
substantial impacts on students’ experiences in some areas, but no impact in others. 

Educational Experiences 

Table 3.5 displays selected results from the survey. As a complement to the analysis of 
Who ’s Next records, students were surveyed about their study habits and their use of student 
services available at HCC. 22 Students in both the program and control groups indicated that they 
had spent an average of approximately nine hours studying for their last paper, exam, or project. 
However, students in the two groups differed in where they had sought assistance: program 
group students were 7.5 percentage points more likely than their control group counterparts to 
have asked a tutor for help (20 percent of program group students versus 13 percent of control 
group students). This suggests that the program changed the rate at which students made use of 
certain academic support services. Test anxiety was mentioned as an issue during student focus 
groups; program group students may have been using tutors as a means to address this issue. 
Despite being more likely to seek assistance from tutors, program group students felt as pre- 
pared as control group students for their most recent final paper, exam or project, with 60 to 65 
percent of both groups indicating that they felt “quite a bit” or “extremely well” prepared. It is 
unknown whether the program group students’ increased use of tutors as a source of assistance 
translated into an improved result on their papers, exams, or projects. 

As expected from the analysis of Who ’s Next data, program group students reported 
that they were much more likely to have visited the Math Labs than control group students: 94 
percent of program group students reported ever having visited the Math Labs, compared 


21 For example, as of the spring 2012 semester, the fall 2010 cohort had completed the program, while the 
fall 2011 cohort continued to receive the scholarship. Results discussed in this section were analyzed for each 
cohort separately, in addition to being analyzed for all cohorts together. Results for each cohort individually 
were generally similar. 

22 The survey questions discussed in this paragraph were not specific to math; the survey asked students 
about their overall study habits, use of resources, and academic preparedness across all subject areas together. 
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The Performance-Based Scholarship Demonstration 
Table 3.5 

Student Educational Experiences 


Hillsborough Community College 


Sample 

Program 

Control 

Standard 

Outcome 

Size 

Group 

Group 

Difference 

Error 

Currently attending or was ever enrolled in 






postsecondary institution beyond high school, 






after random assignment (%) 

847 

96.1 

94.4 

1.8 

1.5 

Total hours spent on most recent final paper/exam/project 

842 

8.6 

8.8 

-0.2 

1.0 

Sought help on most recent final paper/exam/project from' 1 (%) 






Tutor 

842 

20.0 

12.5 

y 5 *** 

2.7 

Instructor 

842 

22.9 

22.0 

0.9 

3.1 

Other students in the class 

842 

26.1 

25.5 

0.7 

3.2 

People not in the class 

842 

17.6 

17.6 

0.0 

2.8 

No one 

842 

39.7 

44.4 

-4.7 

3.6 

Felt quite a bit or extremely well prepared for 






most recent final paper/exam/project (%) 

845 

65.1 

62.4 

2.7 

3.5 

Hours spent studying for math per day (last 7 days) 

844 

1.5 

1.4 

0.0 

0.2 

Ever visited an HCC Math Lab (%) 

846 

94.0 

74.7 

JQ 4 *** 

2.3 

Usual activity in the Math Lab“ (%) 






Participate in one-on-one tutoring 

845 

49.9 

39.2 

|Q J *** 

3.6 

Participate in group help session 

845 

17.5 

13.4 

4.1 

2.7 

Use online math help resources 

845 

76.4 

52.9 

23 5 *** 

3.3 

Something else 

845 

8.7 

10.2 

-1.5 

2.1 

No usual activity or never visited Math Lab 

845 

6.4 

26.0 

-19 7 *** 

2.4 

Of those who participated in one-on-one tutoring (%) 






Found it helpful or very helpful 

384 

89.8 

86.4 



Of those who participated in group session (%) 






Found it helpful or very helpful 

135 

89.5 

80.0 



Of those who participated in online help (%) 






Found it helpful or very helpful 

569 

89.1 

87.2 



Sample size 

847 

549 

298 




SOURCE: MDRC calculations using responses from the Performance-Based Scholarship Hillsborough Community 
College survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction. National Center for Education Statistics risk factors, and selected pre-random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Outcomes shown in italics are nonexperimental. Statistical significance tests are not conducted on 
nonexperimental outcomes. 

Missing values are not included in individual variable distributions. 

“Distributions may not add to 100 percent because categories are not mutually exclusive. 
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with roughly 75 percent of control group students. 23 Program group students were also more 
likely than control group students to have participated in one-on-one tutoring and were more 
likely to have used online math help. Among both program and control group students who 
used various Math Lab services, more than 80 percent found each of the services to be helpful. 

Employment 

Students were also surveyed about their level of employment. One hypothesized effect 
of perfonnance-based scholarships was that they would allow students to reduce the number of 
hours they were working during the semester and focus more of their attention on academics 
instead. This does not appear to have happened — the proportion of students who reported 
they were working as of the time of the survey, the number of hours they were working, and 
the proportion of students who reported being “financially stressed” was unchanged between 
the program and control groups. (See Appendix Table B.3.) This suggests that the amount of 
the scholarship being offered and the frequency with which program group students received 
the award were not great enough to change the employment situation of program group 
students on average. 

Motivation 

Last, students were surveyed about their motivation to do academic work. As with any 
incentive program, there was some concern that offering students awards in exchange for 
academic perfonnance would cause them to become more “extrinsically” motivated (that is, 
motivated by the promise of scholarship money, rather than by intellectual interest or other 
reasons). Student survey results suggest that this did not occur: students in both the program and 
control groups reported a comparable mix of intrinsic and extrinsic motivation. (See Appendix 
Table B.3.) 

As noted above, MAPS was designed to increase students’ motivation to enroll in and 
complete their math courses and continue to enroll in the next course in the sequence. In focus 
groups, program group students reported that they came into their developmental math 
courses motivated to succeed; however, many noted that the program did “push” them to 
work a bit harder to make sure they met the academic benchmark, which none felt was 
particularly high. Many students expressed that they were already motivated to earn the “C” 
grade required to pass the course and move up to the next course. In a program student focus 


~ 3 This is somewhat higher than the proportion of the control group that was recorded in Wiw ’s Next data 
as having visited the Brandon and Dale Mabry Math Labs. (See Table 3.4.) The difference may be due to a 
variety of factors, including inaccurate recollections by students, problems with the Who ’s Next system, or 
differences in the subset of the control group who responded to the survey. 
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group, the general reaction to the scholarship offer was, “Why not get paid for doing some- 
thing that you have to do anyway?” 

Overall, MAPS students in focus groups described the textbook bonus for earning an 
“A” or “B” grade as a better motivator to improve their grade in math compared with the 
scholarship for earning a “C.” These students expressed frustration with the rising prices of 
college textbooks, a common theme for low- and moderate- income students. 24 Students report- 
ed that their math textbooks usually cost more than $100. Many MAPS students said they 
thought about the bonus benchmark every day in class. As one student put it, “The MAPS 
program is making me do wonders. . ..I would have settled for a C, but I’m fighting for an A.” 

When asked what they thought about the opportunity to earn a MAPS award, control 
group students were split between those who felt that they would have focused and tried harder 
in their math classes had they been selected for the program and those who did not think that the 
scholarship would have changed their behavior. 

Summary 

This chapter described how MAPS was implemented, including the roles of college staff and 
differences in experiences between students in the program and control groups. Overall, the 
program was implemented well, with a high level of fidelity to the model. Student scholarship 
payments were disbursed accurately with few errors, and program group students experienced 
MAPS as a notable difference from regular college services. The next chapter looks at the 
impact of MAPS on students’ academic outcomes. 


24 The Advisory Committee on Student Financial Assistance (2007). 
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Chapter 4 

Program Effects on Academic Outcomes 


This chapter of the report presents findings on the Mathematics Access Perfonnance Scholarship 
(MAPS) program’s effects on students’ academic progress at Hillsborough Community College 
(HCC) during the first two years after they entered the study. The results represent the estimated 
effect of the opportunity to participate in MAPS on (1) students’ academic progress in math 
(both developmental and college-level) and (2) students’ overall academic progress in school. 

The program’s effects, or impacts, are estimated by comparing the outcomes of all stu- 
dents who were randomly assigned to the program group with the outcomes of all students who 
were randomly assigned to the control group. 1 Random assignment results in two groups of 
students who are similar at the outset of the study, both with respect to their observable charac- 
teristics (for example, gender, age, and race) as well as unobservable characteristics (for 
example, tenacity and motivation). As a result, subsequent substantial differences in outcomes 
between the two groups can be attributed with confidence to the opportunity to participate in 
MAPS, rather than to preexisting systematic differences between the two groups. 


Main Findings 

After two years, students who were offered the opportunity to participate in MAPS performed 
better, particularly in mathematics, than they would have in the absence of the program. 
Specifically: 

• MAPS helped move students further along in the math course sequence. 

Compared with their control group counterparts, MAPS students were 1 1 percentage 
points more likely at the end of two years to complete a college-level math course or Interme- 
diate Algebra as their highest-level math course. In terms of math credits, this translates 
to an estimated average effect of 0.7 math credit, or a 15 percent increase in math credit 
accumulation. 

• There is evidence that MAPS improved students’ overall academic pro- 
gress as measured by total credit accumulation. This occurred, in part, 
because students made greater progress in mathematics. 


'Students who were not enrolled at HCC during the follow-up period are treated as nonenrollees who 
earned zero credits. 
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The program’s positive effect on progress in math did not come at the expense of other 
coursework. There is no evidence that MAPS had a negative effect on students’ progress in 
their non-math courses. The program’s estimated effect on total credit accumulation, including 
both math and non-math credits, is statistically significant after one semester, one year, and one 
year and one semester. After two years, students who were offered MAPS earned 1.6 more total 
credits, a 7 percent increase, than their control group counterparts, although this difference is not 
statistically significant. Taken together, this evidence suggests that MAPS improved students’ 
overall academic progress. 

• The program had no discernible impact on students’ retention. 

Over the first four semesters, MAPS had no statistically significant impact on students’ 
enrollment rates. That is, program group students were not significantly more likely to reenroll 
in subsequent semesters than control group students. 

• Analyses suggest that MAPS may be more effective for financially inde- 
pendent students and for students who entered the study more than a 
year after graduating from high school or receiving a General Educa- 
tional Development (GED) certificate. 

There is evidence that suggests that MAPS was more effective for financially inde- 
pendent students. 2 It is hypothesized that since these students are responsible for their own 
financial situation, a monetary incentive is more salient and provides a greater motivation to 
them. This finding will be strengthened if it can be replicated at other colleges that are 
implementing performance-based scholarships. MAPS also appears to have been more 
effective for students who entered the study a year or more after graduating from high school 
or earning a GED certificate. This subgroup is intended as a proxy for students who have been 
out of the math classroom for a while. HCC staff members hypothesized that MAPS might be 
most effective for such students because they may have experienced learning loss; thus, 
encouragement to take math and an incentive to use support services could prove particularly 
beneficial to these students. Results provide support for this theory. Notably, though, the 
subgroup based on financial independence and the subgroup based on delayed college 
enrollment have significant overlap, so these findings may not be independent. 

The above results are described in detail below. 


2 Students were defined as financially independent if they met any of the following criteria as of their ran- 
dom assignment into the study: they were age 24 years or older, they were married, or they had one child or 
more. This imperfect proxy for financial independence is an adaptation of the definition used for the Free 
Application for Federal Student Aid (FAFSA), based on the data available to the research team. 
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Description of Key Outcomes 

Before delving into the findings, it is important to understand the outcome measures that were 
used to assess the program’s effectiveness. The analyses of the academic effects of MAPS 
emphasize two confirmatory domains of progress : 3 progress in math and overall academic 
progress. 

Math Progress 

MAPS was designed to provide an incentive for students to take and complete their re- 
quired math courses, starting with Beginning Algebra, then progressing through Intermediate 
Algebra, and eventually completing at least one college-level math course. Faster progress in 
math could be achieved by getting a greater proportion of students to attempt (and pass) math 
courses or by increasing pass rates among students who attempt math courses. Math progress is 
examined by looking at student success in key math classes in and beyond the developmental 
course sequences. A summary measure of math progress — the number of credits students earn 
in math courses, including both developmental and college-level courses — is also used . 4 
Improved math progress and completion was the main goal of MAPS. 

Overall Academic Progress 

Also of importance is whether MAPS improves students’ overall progress toward a de- 
gree. MAPS could improve students’ overall progress by improving their progress in math and 
without impeding progress in non-math courses. MAPS could also improve students’ overall 
progress by improving their progress in math, which in turn could have positive spillover effects 
in other subject areas. It is also plausible, although not intended, that MAPS could have a 
positive effect on students’ math progress and a negative effect on progress in other subject 
areas if, for example, students took fewer non-math courses or focused their time and attention 
on math to the detriment of their other courses, yielding no overall benefit, simply a substitution 
of credit types. Thus it is critical to consider students’ academic progress in non-math courses 
and their overall academic progress when interpreting the effect of MAPS on math progress. 
There is no perfect measure of overall academic progress, only useful proxies. This report’s 


3 Confirmatory outcomes are prespecified tests of the study’s central hypothesis. 

4 Math credits earned was prespecified as the confirmatory indicator of progress in math. For an expla- 
nation of the difference between confirmatory and exploratory outcomes, see Schochet (2008). Examining 
program effects on math progress using math course completions or math credit accumulation provides 
qualitatively similar findings. Consequently, course completions are frequently presented for ease of 
exposition. 
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proxy is calculated by summing together all credits earned, including math and non-math 
credits that are either developmental or college-level credits. 


Detailed Findings 

Presented first is the effect of MAPS on the full research sample during the first two years after 
students entered the study. 5 Full sample results begin with students’ progress in specific math 
courses (shown in Table 4.1) and then describe students’ math, non-math, and overall credit 
accumulation (shown in Table 4.2). Findings are presented for the full sample, after which the 
results are disaggregated to understand whether the program’s effects varied for different 
student subgroups. (Refer to Box 3.1 in Chapter 3 for information about how to read the tables 
presented in this report.) 

Table 4.1 depicts students’ progress in math courses during the first two years after 
students entered the study. Results are arranged in four panels: The first panel presents 
outcomes after the first semester in which students were part of the study — their first 
program semester, either a fall or a spring term. The second panel presents cumulative results 
after students’ first full academic year in the study; this includes a fall, spring, and summer 
semester for all students, regardless of whether they entered the study in the fall or spring 
semester. The third panel presents cumulative findings after the first full academic year plus 
one semester. This represents the end of the MAPS program semesters, when scholarship 
dollars were available to students based on meeting the MAPS benchmarks. Finally, the last 
panel presents cumulative findings after two full academic years in the study, including two 
fall, spring, and summer semesters. The two-year mark includes one semester and one 
summer session after MAPS was complete. 6 Figure 4.1 provides a visual display of these time 
periods for the three cohorts in the evaluation. 7 


5 The analyses presented in this chapter include all program group students and all control group students, 
unless otherwise noted. That is, the analyses are what researchers refer to as intent-to-treat (ITT), as described 
in principle in Bloom (1984). 

6 For the third cohort of students, data were not available for the final summer of the two-year follow-up. 
For these students, the final panel includes only fall, spring, summer, fall, and spring. Since few credits are 
attempted and earned during the second summer of study, it is expected that this has a negligible influence on 
the presented data. 

different time periods are used in the analyses presented in Chapters 3 and 4. In Chapter 3, spring and 
summer semesters are always combined when counting semesters following random assignment. In Chapter 4, 
spring and summer semesters are treated separately for the spring 2011 cohort in order to calculate more 
comparable estimates of program effects across all cohorts. Figures 3.1 and 4.1 illustrate the differences 
between the time periods used in the two chapters. 
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The Performance-Based Scholarship Demonstration 
Table 4.1 

Math Course Progress (Cumulative) 


Hillsborough Community College 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First semester 

Beginning Algebra 

Registered 

92.5 

82.5 

10.0 *** 

2.0 

Passed 

55.9 

45.7 

10.2 *** 

3.1 

Received "A" or "B" 

40.4 

33.0 

7.4 ** 

3.0 

Intermediate Algebra 

Registered 

0.0 

0.0 

0.0 


Passed 

0.0 

0.0 

0.0 


Received "A" or "B" 

0.0 

0.0 

0.0 


College-level math 

Registered 

0.1 

0.0 

0.1 


Passed 

0.0 

0.0 

0.0 


Received "A" or "B" 

0.0 

0.0 

0.0 


First vear 

Beginning Algebra 

Registered 

95.0 

91.2 

3.8 ** 

1.6 

Passed 

65.5 

59.8 

5.7 * 

3.1 

Received "A" or "B" 

46.3 

43.0 

3.3 

3.1 

Intermediate Algebra 

Registered 

55.0 

43.9 

1 *** 

3.1 

Passed 

39.0 

28.1 

10.9 *** 

3.0 

Received "A" or "B" 

25.7 

18.2 

7 4 *** 

2.6 

College-level math 

Registered 

13.4 

8.3 

5 j ** 

2.0 

Passed 

10.0 

6.7 

3.3 * 

1.8 

Received "A" or "B" 

5.3 

4.6 

0.7 

1.4 

First vear and one semester 

Beginning Algebra 

Registered 

95.6 

91.9 

3 7 ** 

1.5 

Passed 

67.9 

62.7 

5.2 * 

3.0 

Received "A" or "B" 

47.4 

45.0 

2.4 

3.1 

Intermediate Algebra 

Registered 

61.0 

52.6 

g ^ *** 

3.1 

Passed 

45.0 

34.6 

10 4 *** 

3.1 

Received "A" or "B" 

27.0 

21.2 

5 7** 

2.7 

College-level math 

Registered 

34.7 

25.0 

g 7 *** 

2.9 

Passed 

26.2 

20.3 

5.9 ** 

2.7 

Received "A" or "B" 

13.5 

13.7 

-0.2 

2.2 


(continued) 
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Table 4.1 (continued) 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First two vears 

Beginning Algebra 

Registered 

95.8 

92.4 

3.4 ** 

1.5 

Passed 

69.0 

64.5 

4.5 

3.0 

Received "A" or "B" 

48.2 

45.7 

2.4 

3.1 

Intermediate Algebra 

Registered 

63.0 

56.1 

6.9 ** 

3.1 

Passed 

48.4 

37.8 

JQ 7 *** 

3.1 

Received "A" or "B" 

28.0 

22.2 

5.8 ** 

2.7 

College-level math 

Registered 

41.3 

32.1 

g 2 *** 

3.1 

Passed 

31.9 

26.4 

5.6 * 

2.9 

Received "A" or "B" 

16.9 

17.7 

-0.7 

2.4 

Highest level of math passed 

None 

30.8 

35.2 

-4.4 

3.0 

Beginning Algebra 

20.4 

27.0 

-6.5 ** 

2.7 

Intermediate Algebra 

16.8 

11.4 

5.4 ** 

2.3 

College-level math 

31.9 

26.4 

5.6 * 

2.9 

Sample size (total = 1,075) 

674 

401 




SOURCE: MDRC calculations from Hillsborough Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre -random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Beginning Algebra is a 4-credit developmental course. Intermediate Algebra is worth 3 college-level credits. 
College-level math courses are worth 3-5 college-level credits. 

To pass Beginning Algebra and Intermediate Algebra, students must earn a grade of "C" or higher. To pass 
college-level math courses, students must earn a grade of "D" or higher. 

Students may have registered for multiple college-level math courses in the same semester. Students are 
flagged as "passed" as long as they passed one college-level math course. 


Math Progress 

First Semester 

Recall that all study participants needed to enroll in Beginning Algebra. This is the first 
course for which program group students could receive scholarship payments — $100 for 
enrolling in the course and $500 for visiting the Math Lab and completing the course with a 
grade of “C” or better. In the absence of MAPS, 82.5 percent of control group students regis- 
tered for Beginning Algebra. With MAPS, program group students registered at a rate of 92.5 
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The Performance-Based Scholarship Demonstration 

Figure 4.1 

Time Periods Used in Chapter 4 
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percent. The difference, 10.0 percentage points, represents the estimated effect of offering 
MAPS on students’ likelihood of registering for Beginning Algebra during the first program 
semester. Tying this back to the program’s intent to decrease learning loss by getting students to 
enroll immediately in math courses, in the first semester the program succeeded in getting a 
group of students who otherwise would have put off their first math class to take it immediately. 
This initial effect on enrollment is critical since a program that increases the percentage of 
students who attempt Beginning Algebra can also have an impact on the percentage of students 
who complete the course, even if pass rates are unaffected. 

In this case, 55.9 percent of all program group students passed Beginning Algebra com- 
pared with 45.7 percent of all control group students. 8 The estimated effect, 10.2 percentage 
points, corresponds to a 22 percent increase above the base of 45.7 percent. The program’s 
estimated effect on passing Beginning Algebra was a result of both increased attempts and 
increased rates of passing among those who attempted the course. Among program group 
members who took the course, 60 percent passed. Among control group members who took the 
course, 55 percent passed. 9 The program’s first semester effect on pass rates in Beginning 
Algebra is important because progressing in math is the main focus of MAPS. That said, there 
is still much room for improvement — 44 percent of program group students had not passed 
Beginning Algebra by the end of their first semester. 

One way to think about the impact on students’ completing Beginning Algebra is to 
consider how many additional students are estimated to have completed the course as a result of 
MAPS. In this evaluation, there are 674 program group members, and the estimated effect of 
MAPS on completing Beginning Algebra is 10.2 percentage points. Thus, it is estimated that 
MAPS caused 69 more program group students (674 multiplied by 0.102) to pass Beginning 
Algebra in their first semester than would have been expected to pass had those 674 students not 
been offered MAPS. 

Not only did MAPS result in more students registering for and passing Beginning Al- 
gebra, it also increased rates of receiving an “A” or “B” in Beginning Algebra by 7.4 percentage 
points — another outcome for which the program provided a direct incentive. 


8 The denominator in these calculations includes all study participants, including those who did not enroll 
in the Beginning Algebra course. 

9 Program and control group differences in pass rates among those who attempted the course are not exper- 
imental and are not shown in the table. They can be calculated as 55.9/92.5 for program group members and 
45.7/82.5 for control group members. The different pass rates between program and control group members 
may reflect the effectiveness of MAPS or differences in the types of students who took the course (or both). 
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First Year 


The second panel in Table 4. 1 shows students’ cumulative progress through an academ- 
ic year in MAPS, comprising a fall, spring, and summer term. Looking across the math courses, 
a clear picture unfolds: students in MAPS made significantly more progress through the math 
course sequence than did their control group counterparts. With respect to Beginning Algebra, 
compared with the magnitude of the estimated effects in the first panel of the table, the estimat- 
ed effects are smaller in the second panel, a result of the fact that more control group students 
passed this course in the interim, a time period when a greater percentage of program group 
students had already moved on to the next course. The positive impact estimate (5.7 percentage 
points) on passing Beginning Algebra shows that the control group still did not fully catch up on 
this outcome. Meanwhile, during the time that control group students were narrowing the gap in 
passing Beginning Algebra, 11.1 percentage points more program group students took Interme- 
diate Algebra, and 10.9 percentage points more passed Intermediate Algebra, showing that 
MAPS students made significantly more progress in the next stage of the course sequence than 
they would have in the absence of the program. Finally, after one year, including a fall, spring, 
and summer semester, more students who were offered MAPS had registered for (13.4 percent) 
and passed (10.0 percent) college-level math than their control group counterparts (8.3 percent 
and 6.7 percent, respectively). 

First Year and One Semester 

The third panel in Table 4.1 shows students’ cumulative progress in math through an 
entire academic year in MAPS, comprising a fall, spring, and summer term, plus one additional 
semester (either a fall or a spring semester, depending on the cohort). Comparing the second 
and third panels provides a sense of students’ progress during the additional semester, the final 
semester of the program. With respect to passing Beginning Algebra and Intermediate Algebra, 
the program effects were fairly stable at this point. With respect to college-level math, by the 
end of MAPS, 26.2 percent of program group members passed a college-level math course 
compared with 20.3 percent of control group members, for an estimated effect of 5 .9 percentage 
points. After one year and one semester, MAPS students maintained an advantage in their math 
course sequence. 

First Two Years 

The final panel of Table 4. 1 shows the cumulative effect of the opportunity to partici- 
pate in MAPS after two full years. Comparing the third and fourth panels shows little change in 
terms of both outcome levels and impact estimates, with the exception that the outcome levels 
in college-level math improved for both program and control group members. The overall story 
remains the same — program group students progressed further in their math course sequence. 
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The final portion of the fourth panel presents one additional summary measure: the 
highest level of math passed. This provides a sense of the distribution of where students were in 
their math progress at the end of the follow-up period. After two years, 26.4 percent of control 
group members had passed a college-level math class compared with 31.9 percent of program 
group members. MAPS is estimated to have increased students’ chances of completing college- 
level math by 5.6 percentage points. Another way to look at math progress is to note that MAPS 
is estimated to have moved 1 1.0 percent of students (5.4 plus 5.6) from not having completed 
any math or having only completed Beginning Algebra to having completed Intermediate 
Algebra or having completed college-level math. 10 The program had a clear, positive effect on 
students’ math progress during the two year follow-up; nonetheless, the majority of MAPS 
students (51.2 percent) still had not completed Intermediate Algebra, let alone a college-level 
math course, two years after entering the study. 

Credit Accumulation 

Table 4.2 depicts students’ cumulative credits attempted and earned during the first two 
years after entering the study. Credits are shown in total as well as disaggregated into math and 
non-math credits. The four panels in Table 4.2 represent the same time frames as in Table 4. 1 . 11 

First Semester 

In terms of math credits, the program’s estimated effect on both credits attempted and 
earned is 0.4 credit. Program group students earned an average of 2.3 math credits whereas 
control group students earned an average of 1.8 credits. 12 The estimated program effect repre- 
sents a 22 percent increase in math credits earned resulting from MAPS. This corresponds to the 
results presented in Table 4.1, which showed that MAPS students took and completed Begin- 
ning Algebra at a higher rate than their control group counterparts. 

Although MAPS was designed to improve math progress, it is important to consider 
whether progress in math came at the expense of progress in non-math courses. The first panel 
of Table 4.2 shows no discernible evidence that MAPS decreased students’ progress in their 
non-math courses, as illustrated by the positive, though statistically insignificant, estimated 
effect on non-math credits earned. (The estimated effect is 0.3 non-math credit earned.) Com- 
bining math credits and non-math credits yields total credits, a proxy for students’ overall 
academic progress. The program’s estimated effect on total credits earned in the first semester is 


10 This assumes that MAPS did not have a negative effect on any students’ outcomes. 
"Appendix Table C.l provides more detailed information that corresponds to Table 4.2. 
12 Rounding may cause slight discrepancies in sums and differences. 
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The Performance-Based Scholarship Demonstration 
Table 4.2 

Credits Attempted and Earned (Cumulative) 


Hillsborough Community College 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First semester 

Total credits attempted 

10.7 

10.5 

0.3 

0.2 

Math 

3.7 

3.3 

Q /J. *** 

0.1 

Non-math 

7.0 

7.1 

- 0.1 

0.2 

Total credits earned 

7.5 

6.8 

0 7** 

0.3 

Math 

2.3 

1.8 

o 4 *** 

0.1 

Non-math 

5.3 

5.0 

0.3 

0.2 

First vear 

Total credits attempted 

21.6 

20.7 

0.9 

0.6 

Math 

6.8 

5.9 

0 9 *** 

0.2 

Non-math 

14.8 

14.8 

0.0 

0.5 

Total credits earned 

15.0 

13.7 

1.3 ** 

0.6 

Math 

4.1 

3.5 

0 7 *** 

0.2 

Non-math 

10.9 

10.3 

0.6 

0.5 

First vear and one semester 

Total credits attempted 

27.6 

26.3 

1.4 * 

0.8 

Math 

8.4 

7.3 

j j *** 

0.2 

Non-math 

19.3 

19.0 

0.3 

0.7 

Total credits earned 

19.2 

17.6 

1.5 * 

0.9 

Math 

5.0 

4.3 

Q g *** 

0.3 

Non-math 

14.1 

13.4 

0.8 

0.7 

First two vears 

Total credits attempted 

33.2 

31.6 

1.7 

1.1 

Math 

9.5 

8.5 

1 0 *** 

0.3 

Non-math 

23.7 

23.1 

0.6 

0.9 

Total credits earned 

23.1 

21.5 

1.6 

1.1 

Math 

5.6 

4.9 

07 ** 

0.3 

Non-math 

17.5 

16.6 

0.9 

0.9 

Sample size (total = 1,075) 

674 

401 




SOURCE: MDRC calculations from Hillsborough Community College transcript data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre-random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


57 






0.7 credit, a statistically significant effect representing a 10 percent increase over the control 
group base of 6.8 credits. Fifty-nine percent of the estimated impact on total credits earned was 
a result of additional math credits earned. 

First Year 

In terms of cumulative math credits, the program’s estimated effect on credits attempted 
and earned is 0.9 and 0.7, respectively, at the one-year mark. By the end of the first year, MAPS 
students earned an estimated 20 percent more math credits than they would have in the absence 
of the program. 

The program’s estimated effect on total credits earned after the first year is 1.3 credits, 
representing a 9 percent increase over the control group base of 13.7 credits. As before, there is 
no evidence that MAPS had a deleterious effect on students’ non-math progress, as revealed by 
the larger estimated effect on total credit accumulation than the estimated effect on math credit 
accumulation alone. 

First Year and One Semester 

After one year and one semester, the estimated effect of MAPS on math credit accumu- 
lation remained about the same, at 0.8 credit, indicating that at this point program group 
students and their control group counterparts were progressing at about the same rate. As a 
result, program students remained ahead of control group students in their progress in the 
overall course sequence. The estimated effect on total credits earned after the first year and one 
semester increased to 1 .5 credits, slightly above that of the one-year mark. 

First Two Years 

At the end of two years, the effect on math credit accumulation was 0.7 credit. The fact 
that this impact remained remarkably stable from the one-year mark onward indicates that early 
on in MAPS, program group students earned more math credits than did their control group 
counterparts. Then, in later semesters, the program and control groups each earned math credits 
at about the same rate. The control group did not catch up, nor was the gap widened; rather, the 
program’s early positive effect on math credit accumulation was maintained over two years. 
Appendix Table C.2 provides information similar to that depicted in Table 4.2, except that it 
depicts each semester’s results separately and noncumulatively. Results in this format support 
the conclusion above — positive early effects are maintained, but do not grow — and they may 
be of interest to readers of past MDRC reports that frequently show results in this format. 

With respect to non-math credits, at the end of two years, control group students earned 
an average of 16.6 credits and program group students earned an average of 17.5 credits. While 
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the positive estimated effect of 0.9 credit is not statistically significant, it certainly does not 
provide any evidence that MAPS had a deleterious effect on non-math credit accumulation. 

Finally, after two years, control group students earned an average of 21.5 total credits 
and program group students earned an average of 23.1 credits. Figure 4.2 plots cumulative 
credits earned over time, providing a visual depiction of this outcome measure. Although not 


The Performance-Based Scholarship Demonstration 
Figure 4.2 

Total Credits Earned After Two Years 
Hillsborough Community College 



SOURCE: MDRC calculations from Hillsborough Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre-random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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statistically significant, the program’s estimated effect on total credits earned after two years is 
1.6 credits, with a 90 percent confidence interval ranging from -0.3 to 3.5 total credits. The lack 
of statistical significance suggests that caution should be used in claiming that MAPS had a 
positive effect on total credit accumulation, and thus overall academic progress, at the end of the 
study period. However, despite the lack of statistical significance at the final time point, the 
claim seems warranted because (1) the estimated effect of MAPS on cumulative total credits 
earned is positive and statistically significant during all three preceding time points, (2) the 
estimated effect on total credit accumulation increased at each successive time point, 13 and (3) 
there is clear evidence that MAPS had a positive effect on math credit accumulation and no 
discernible evidence that MAPS had a negative effect on non-math credit accumulation. Taken 
together, this evidence suggests that MAPS improved students’ overall academic progress, in 
part because of its positive effect on progress in math — 46 percent of the estimated effect on 
total credits is attributed to math credits. 

Notably, despite the program’s clear positive effect on math progress and evidence of 
an effect on overall progress, there is no evidence of a discernible effect on persistence, as 
shown in Appendix Table C.2. 

Subgroup Findings 

In addition to estimating the overall effectiveness of MAPS, the program’s effects for differ- 
ent types of students are of interest. Hidden in the overall average results presented above is 
the possibility that MAPS may be more or less effective for certain types, or subgroups, of 
students. 

Two subgroups of students were prespecified as being of particular interest for sub- 
group analyses because of logical hypotheses that MAPS may be more effective for one group 
than another. The first hypothesis is that MAPS may be more effective for students who are 
financially independent. The theory is that a monetary incentive, on average, may be a greater 
and more salient motivator for students who are primarily responsible for their own financial 
situation. The importance of and relationship to money may be different for students who are in 
charge of their own fiscal matters. Consequently, the response of, and effects on, students who 
are offered a monetary incentive may vary based on this characteristic. 


13 It is somewhat counterintuitive that the estimated impact at the end of two years can be greater than the 
estimated impact at the end of the first year plus one semester, and yet the latter is statistically significant and 
the former is not. The reason for this is that the standard error, a measure of uncertainty of the estimated 
impact, is larger at later time points because the variance of cumulative credits earned increases over time. 


60 



The second hypothesis, generated by staff members at HCC, is that MAPS may be 
more effective for students who have not been in a math classroom for some time. Such 
students may have experienced significant learning loss. Thus, an incentive to get back into 
mathematics and to take advantage of the Math Labs for support might prove especially 
effective for these students, compared with students who did not experience any delay between 
taking their previous math classes and enrolling in the program. Unfortunately, no measure of 
“time since taking last math class” is readily available at baseline, so, instead, two proxies are 
used. The first one is a binary indicator of whether students enrolled in the evaluation more than 
a year following the date when they received their high school diploma or GED certificate. 14 
The second one is a binary indicator of whether a student was age 20 or older upon entering the 
study. While these measures do not correspond perfectly with the construct of interest, they 
serve as proxies. 

In addition to exploring the program’s effects on these subgroups, which were selected 
based on theory, the study examines the effects based on students’ race and gender. Although a 
variety of hypotheses can be imagined for why MAPS might have differential effects based on 
race and gender, these subgroups are explored primarily because of their political importance. 
Policymakers and researchers are often interested in whether program effects vary by race and 
gender owing to historical disadvantages experienced by different groups and achievement gaps 
with respect to education. 

Table 4.3 summarizes the program’s cumulative estimated effects on math credits 
earned at the end of two years by the five different subgroups: (1) financial dependence, (2) 
greater than one year since receipt of a high school diploma or GED certificate, (3) age 20 
years or older, (4) gender, and (5) race/ethnicity. 15 The results suggest that the effectiveness of 
MAPS varied: MAPS may be particularly effective for certain types of students and have 
minimal or no effect on others. For example, as hypothesized, financially independent 
students appear to have benefited more from MAPS than financially dependent students 


14 This is an imperfect measure because many students entered the MDRC study as continuing students, so 
although they may have entered the study a year or more after graduating from high school or earning a GED 
certificate, they may have been enrolled in college during that year and they may even have taken a math class 
during that time period. 

l5 The first two subgroups listed were prespecified as confirmatory in an analysis plan that was developed 
before examining the results. These two groups were prespecified based on an assessment concluding that the 
theory about possibly differential effects was stronger for these groups. The general approach of prespecifying 
a very small number of subgroups as confirmatory is an attempt to reduce the multiple hypotheses testing 
problem — that is, the problem that conducting many hypothesis tests can lead to spurious findings. Multiple 
hypothesis testing adjustments were not conducted. 
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The Performance-Based Scholarship Demonstration 
Table 4.3 

Variation in Program Effects on Number of Math Credits Earned 
After Two Years, by Student Characteristics 


Hillsborough Community College 








Difference 

Subgroup 


Sample 

Program 

Control 


Standard 

Between 

Difference 

Characteristic 

Size 

Group 

Group 

Difference 

Error 

Subgroups 

P-value 

Financial dependence 11 

Independent 

652 

6.0 

4.9 

1 q *** 

0.4 

tt 

0.0474 

Dependent 

368 

5.0 

5.3 

-0.3 

0.6 



Greater than one vear since hish school/GED 

Yes 

844 

5.8 

4.6 

1 1 *** 

0.3 

ftf 

0.0008 

No 

192 

4.5 

6.3 

-1.8 ** 

0.8 



Ace 20 vears or older 

Yes 

824 

5.9 

4.8 

1 1 *** 

0.3 

tt 

0.0145 

No 

251 

4.8 

5.5 

-0.7 

0.7 



Gender 

Male 

362 

5.6 

4.4 

1.2 ** 

0.6 


0.3824 

Female 

713 

5.7 

5.1 

0.6 * 

0.4 



Race/ethnicitv b 

Flispanic 

325 

5.2 

5.5 

-0.3 

0.5 

tt 

0.0107 

White 

323 

5.9 

5.7 

0.3 

0.6 



Black 

351 

5.7 

4.0 

| 7 *** 

0.5 



Other 

60 

6.3 

2.1 

4.2 ** 

1.8 
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Table 4.3 (continued) 

SOURCES: MDRC calculations from Hillsborough Community College transcript data and Baseline Information Form data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort-campus interaction, National Center for 
Education Statistics risk factors, and selected pre-random assignment placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups. Statistical significance levels are 
indicated as follows: fff = 1 percent; (f = 5 percent; | = 10 percent. 

Two years of data represents two fall semesters, two spring semesters, and two summer semesters for the fall 20 1 0 and spring 2011 cohorts, and two fall 
semesters, two spring semesters, and one summer semester for the fall 201 1 cohort. 

a Students were counted as financially independent if they met any one of the following criteria as of random assignment: they were age 24 years or older, 
they were married, or they had one child or more. 

b Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. Respondents who said they are not Hispanic and 
chose more than one race are included in the Other category. These respondents, combined with those who said they were Asian, American Indian or Alaska 
Native, or another race/ethnicity, are included in Other. 
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(estimated impacts of 1.0 math credit and negative 0.3 math credit, respectively). Similarly, as 
hypothesized, MAPS had a positive and significant effect on students who graduated from 
high school or earned a GED certificate more than one year before entering the evaluation, 
and this effect is statistically significantly different from the estimated effect for students who 
recently graduated or earned a GED certificate (estimated impacts of 1.1 math credits and 
negative 1.8 credits, respectively). 

Surprisingly, MAPS is estimated to have had a negative effect on the small proportion 
of the sample that recently graduated from high school or earned a GED certificate. This 
group includes only 192 students, a small sample, and the standard error is quite large. 
However, caution may be warranted when considering offering MAPS to this subgroup of 
students. 

MAPS may have also been more effective for students age 20 years or older. Since 
the students in this study who are financially independent are also disproportionately those 
who graduated from high school more than one year before random assignment, and, further, 
are also disproportionately students who are age 20 years or older, these subgroup findings 
are likely related. Indeed, 54 percent of the sample falls into all three groups, while 1 1 percent 
falls into none. A future report synthesizing findings across MDRC’s Performance-Based 
Scholarship Demonstration will examine the issue of differential effects based on financial 
dependence. 

In other key subgroups of interest, analysis finds no evidence that MAPS has a differ- 
ential effect for men compared with women; rather, it has a positive effect for both groups. By 
contrast, analyses suggest that MAPS was most effective for black students and students who 
responded that they were Asian, American Indian or Alaska Native, multiracial, or another 
race/ethnicity (listed as “Other”), compared with those who responded that they were white or 
Hispanic. The evidence of differential effects based on race does not correspond with findings 
from other studies of the effects of performance-based scholarships, and thus caution is urged 
when considering these observed differences. 

The general pattern of subgroup findings is confirmed in Table 4.4, which examines 
the same subgroups with respect to total credits earned. Should the subgroup findings pre- 
sented in this report be confinned in future studies involving financial incentives for commu- 
nity college students, an argument could be made for the careful targeting of financial incen- 
tives toward those students who benefit most from these incentives. For now, these findings 
remain exploratory and warrant further investigation. 
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The Performance-Based Scholarship Demonstration 
Table 4.4 

Variation in Program Effects on Number of Total Credits Earned 
After Two Years, by Student Characteristics 


Hillsborough Community College 








Difference 

Subgroup 


Sample 

Program 

Control 


Standard 

Between 

Difference 

Characteristic 

Size 

Group 

Group 

Difference 

Error 

Subgroups 

P-value 

Financial dependence 3 

Independent 

652 

23.6 

20.7 

29 ** 

1.5 

t 

0.0532 

Dependent 

368 

22.1 

24.1 

- 2.0 

2.1 



Greater than one vear since high school/GED 

Yes 

844 

22.9 

20.0 

29 ** 

1.3 

tt 

0.0156 

No 

192 

22.5 

28.0 

-5.5 * 

3.2 



Ase 20 vears or older 

Yes 

824 

23.2 

20.4 

2 g ** 

1.3 

t 

0.0618 

No 

251 

22.8 

25.4 

-2.5 

2.6 



Gender 

Male 

362 

24.0 

20.1 

3.9 * 

2.2 


0.2330 

Female 

713 

22.8 

22.0 

0.8 

1.3 



Race/ethnicitv b 

Fiispanic 

325 

20.4 

23.1 

-2.7 

2.0 

tt 

0.0317 

White 

323 

23.9 

22.0 

1.8 

2.3 



Black 

351 

25.0 

20.2 

4 g ** 

2.1 



Other 

60 

25.5 

14.3 

11.3 

7.4 




(continued) 



Table 4.4 (continued) 

SOURCES: MDRC calculations from Hillsborough Community College transcript data and Baseline 
Information Form data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre -random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The H-statistic test was used to test for statistically significant differences in impact estimates across 
different subgroups. Statistical significance levels are indicated as follows: fff = 1 percent; ft = 5 percent; f = 
10 percent. 

Two years of data represents two fall semesters, two spring semesters, and two summer semesters for the 
fall 2010 and spring 201 1 cohorts, and two fall semesters, two spring semesters, and one summer semester for 
the fall 2011 cohort. 

a Students were counted as financially independent if they met any one of the following criteria as of random 
assignment: they were age 24 years or older, they were married, or they had one or more children. 

b Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are included in the Other category. 
These respondents, combined with those who said they were Asian, American Indian or Alaska Native, or 
another race/ethnicity, are included in Other. 


Summary 

MAPS led more students to take math courses, complete math courses, and continue to enroll in 
the next course in the sequence. MAPS’s success did not come at the detriment of students’ 
progress in other subject areas; in fact, the program appears to have helped students make 
greater overall progress toward a degree. There is suggestive evidence that MAPS may have 
been more effective for certain types of students — a finding that should be explored in more 
depth in future evaluations of financial incentives for community college students. 
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Chapter 5 

Cost-Effectiveness of MAPS 


This chapter is designed to address questions regarding the decision to invest in the Mathemat- 
ics Access Performance Scholarship (MAPS) program at Hillsborough Community College 
(HCC). The discussion identifies the cost of MAPS, including a detailed examination of the 
various component costs associated with the program. Additionally, the total cost of college for 
students who are eligible for MAPS is compared with the cost of the usual college experience 
without MAPS. Last, the total cost of college for both groups and student success outcomes are 
compared to explore how the performance-based scholarship program at HCC changes the cost 
per outcome achieved — specifically, whether the investment in the program produced more 
desired outcomes per dollar than the usual college services. 1 

The key findings are the following: 

• Scholarship payments made up the majority of program cost. 

The direct cost to operate MAPS at HCC was just under $1,400 ($1,394) per program 
group member. Most of this money (64 percent) was associated with transfers to students in 
the form of scholarship payments and book vouchers. Just over 25 percent of the program 
cost was associated with program administration and 10 percent was associated with in- 
creased Math Lab costs. 

• When the additional costs of educating students are considered, over the 
two years of follow-up, the college invested between $1,394 and $1,863 
more per program group member than it did per control group member. 

This estimate includes the direct cost to operate the program plus an estimate of the cost 
associated with program students’ attempting more college courses, which has a low-end cost 
estimate of $0 and a high-end cost estimate of approximately $469 per program group member. 

• The cost-effectiveness findings on MAPS are mixed and vary by out- 
come. MAPS is able to lower the cost per college-level math course com- 


'lt is possible to lower the cost per outcome achieved while increasing total costs. The analysis compares 
alternative strategies to achieve a designated goal and the result is always relative — that is, one program 
lowers the cost per outcome achieved when compared with another. In this case the program is compared with 
the usual college services. Each alternative requires resources to produce its respective results. The comparison 
aims to highlight which strategy or alternative produces the most positive outcomes per dollar of investment. 
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pletion when compared with the usual college experience but does not 
lower the cost of math and total credits earned. 

The analysis shows that the $1,394 to $1,863 of additional investment in each program 
group member resulted in a 5.6 percentage point increase in the likelihood of completing a 
college-level math course. This impact is large enough that when costs (direct program costs plus 
the cost of credits attempted by students over two years) are tied to the number of college-level 
math course completions, the program lowers the cost per outcome achieved in comparison with 
the usual college services without the program. Specifically, the cost per college-level math 
course completion for the program group is between $126 and $1,594 less (or up to 5 percent 
less) than the cost per college-level math course completion for the control group. The program 
also resulted in students earning 0.7 more math credit and 1.6 more credits overall; these impacts 
were not large enough, however, to lower the cost per math credits or total credits earned. 


Methodology 

This chapter describes the cost of scholarship payments and estimates the cost-effectiveness of 
MAPS at HCC between July 2010 and January 2013. All costs are considered from the perspec- 
tive of the college. College-level spending includes resources from a spectrum of stakeholders 
including students, private donors or foundations, local taxpayers, and state and federal gov- 
ernment. These costs are estimated using college financial information. Since all funds (such as 
tuition paid by students, subsidies from various government entities, and private donations) are 
funneled through the college, this approach provides a good estimate of the total investment 
being made in these community college students. The analysis aims to exclude costs that are not 
part of the “steady state” of operation of MAPS; as a result, start-up and research costs have 
been excluded. 2 See Appendix D for definitions of the temis used throughout this chapter. 

Direct Cost 

The direct cost accounts for the resources that are required to operate MAPS during the 
period of program operation. In this analysis, the period of program operation spanned three 
semesters and one summer semester. As outlined in Figure 5.1, the direct cost of MAPS was 
approximately $1,394 per program group member. The majority (64 percent) of these costs went 
directly to students as transfers in the form of scholarship payments and textbook vouchers. 


2 For purposes of presentation all costs have been classified as either direct, base, or indirect costs. The sum 
of these component costs is the total cost. The difference between the program group’s and control group’s 
total costs is described as the net cost. 
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The Performance-Based Scholarship Demonstration 
Figure 5.1 

Direct Cost per Program Group Member 
Hillsborough Community College 



Total = $1,394 per program group member 


SOURCE: MDRC calculations from MAPS program expenditures and Hillsborough Community College 
scholarship receipt data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. All dollar values have been 
rounded to the nearest whole dollar. 

The time period for this analysis is July 2010 through January 2013. Program costs are based on a steady 
state of operation that excludes research and start-up costs. 


Scholarship payments accounted for 57 percent of the program’s direct cost, with students 
receiving approximately $790 on average. In addition, students received an average of $103 in 
textbook bonuses, accounting for about 7 percent of the program’s direct cost. Program admin- 
istration accounted for just over one-fourth (26 percent) of the program’s direct cost. The bulk 
of program administration was associated with personnel expenses (22 percent of the total direct 
cost) while the remaining 4 percent of program administration covered student tracking and 
other expenses (3 percent and 1 percent of the total direct cost, respectively). Approximately 10 
percent of the program’s direct cost was related to Math Lab staffing expenses. 3 


3 Program group members were required to visit the Math Labs at the Dale Mabry or Brandon campuses to 
earn the scholarship payments for the first two courses in the sequence. As a result of the increased demand 

(continued) 
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Scholarship Costs 

Scholarship costs made up 57 percent of the direct costs incurred by MAPS, with the 
average student receiving $790 in scholarship payments. Therefore, of the potential $1,800 each 
student could have earned in scholarships, 44 percent of the scholarship amount offered was 
ever paid. Higher proportions of payments were made to students taking Beginning Algebra, 
while a smaller proportion went to students in Intermediate Algebra, and a still smaller propor- 
tion to students enrolled in college-level math. This means that a progressively larger portion of 
scholarships offered to students are being left unpaid. This disparity is largely a function of time 
and the nature of the hurdle. As time passes, fewer students enroll, which results in a lower 
percentage of scholarships offered being paid over time. Additionally, some hurdles are less 
likely to be surpassed than others. Understanding the relationship between scholarship amounts 
offered and paid in relation to various outcomes should help similar programs control costs and 
improve overall program efficacy. 

Base Cost 

In order to understand the context of program direct cost, this analysis estimates the cost 
of the usual college services provided to all students, whether or not they are in the program. 
This is also referred to as the base cost. Getting an accurate estimate of this cost can be chal- 
lenging because of data limitations, so this analysis uses the estimated cost of credits attempted 
as a proxy for base costs. This approach assumes that resource use corresponds to the number of 
credits attempted; in other words, a student who attempts more credits is generally associated 
with greater expenditures than a student who attempts fewer credits. 4 To estimate the dollar 
value of credits attempted, the number of credits attempted is multiplied by an estimated cost 
per credit. This cost per credit is estimated by dividing the college’s annual operating budget by 
total instructional activity (credit hours attempted) at the college during the year of interest. The 
average cost per credit yielded by this calculation is then used to estimate the cost of the usual 


that this requirement was expected to create, additional staff capacity was needed. Math Lab staff capacity was 
increased through the training and certification of tutors, the hiring of nine new tutors and increasing the hours 
of four existing staff members at Dale Mabry, and the hiring of seven tutors at the Brandon campus. Over the 
course of the analysis period, the program invested $90,000 in additional Math Lab staff on those two 
campuses to meet the predicted increase in demand. This represents an expenditure of $134 per program group 
member, or 10 percent of the program’s total direct cost. Both program and control group members, as well as 
students who were not in the research sample, may have benefited from the additional staff made possible by 
this expenditure. This cost has been included in the analysis because it was incurred directly as a result of the 
program. 

4 However, it is possible that students use services such as advising and counseling independent of the 
number of courses in which they enroll. 


70 



college experience. 5 This approach is not perfect. One limitation is the assumption that all 
credits attempted have the same cost to the college, which is not necessarily the case. 6 For 
example, science lab courses may be more expensive than English courses. In order to use this 
approach, the analysis assumes that the average cost of an HCC student is representative of the 
average cost of a student in the sample. However, this seems to be a reasonable assumption for 
this analysis because the process of random assignment helps ensure that any differences in the 
cost of credits attempted compared with the average cost would occur equally in both the 
program and control groups. Estimating the base cost helps show how much money is spent to 
educate the typical sample member in the absence of the program. 

Table 5.1 presents all of the costs used in this cost analysis, and calculates a net cost per 
group member. The first row reiterates the direct cost, discussed above. The second row of 
Table 5.1 presents the base cost of credits attempted at HCC. Control group members attempted 
an average of 31.6 credits in the two years of follow-up, for a total cost of credits attempted 
(credits attempted multiplied by cost per credit) of $8,959 per group member. 7 This represents 
the cost of the typical college experience in the absence of this program. 

Indirect Cost 

Indirect cost describes the cost associated with behavioral changes that are a result of 
the program. Tracking the indirect cost helps determine whether the intervention had any 
externalities that affected cost in the long term. 8 This analysis will consider indirect costs under 
two conditions, the first being indirect costs when marginal costs are equal to zero, and the 
second when marginal costs are equal to average cost per credit. A marginal cost equal to zero 
refers to a condition when colleges are fully able to absorb the cost of additional credits at- 
tempted by leveraging existing resources so there is no additional cost to the college (referred to 


5 The total expenditures are divided by the total instructional activity provided at the school in the corre- 
sponding year to calculate the cost per instruction at a college. HCC, like many Florida state colleges, reports 
its yearly instructional activity in contact hours and credit hours. For ease of comprehension, the cost per total 
instructional activity is shown as cost per credit using an Integrated Postsecondary Education Data System 
(IPEDS) conversion where 30 credit hours or 900 contact hours are equivalent to one full-time equivalent 
student (FTE). The cost per credit at HCC was calculated to be $283.85. 

6 “Cosf ’ in this case refers to the amount of resources dedicated to the course by the college; it is not neces- 
sarily connected to the price that students may be paying for that course. 

7 Rounding may cause slight discrepancies in cost calculations because impact data are rounded to one 
decimal place. Rounding may make some numbers appear slightly different from their Chapter 4 counterparts. 
All cost calculations used all four available decimal places in impact data for accuracy. 

s An economic externality is a side effect or consequence of an activity that incurs costs or benefits not 
initially accounted for; in this example, indirect costs capture the externalities of the program by accounting for 
the increase in credits attempted by students over the two year follow-up period. 
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The Performance-Based Scholarship Demonstration 
Table 5.1 

Net Cost of Education per Sample Member 
Hillsborough Community College 


Feature ($) 

Program 

Group 

Control 

Group 

Difference 

Direct cost: cost of primary program components 

1,394 

0 

1,394 

Base cost: cost of credits attempted in the absence of the program 

8,959 

8,959 

0 

Indirect cost: cost of additional credits attempted because of program 




Marginal cost equal to zero 3 

0 

0 

0 

Marginal cost equal to average cost b 

469 

0 

469 

Total cost 




Marginal cost equal to zero 3 

10,353 

8,959 

1,394 

Marginal cost equal to average cost b 

10,822 

8,959 

1,863 


SOURCES: MDRC calculations from student level participation data, program-specific budget data, and 
Hillsborough Community College financial and enrollment data from the Integrated Postsecondary Education 
Data System (http://nces.ed.gov/ipeds/datacenter/). 

NOTES: Rounding may cause slight discrepancies in cost calculations due to the rounding of impact data to 
one decimal place. All cost calculations used all four available decimal places in impact data for accuracy. All 
dollar values have been rounded to the nearest whole dollar. 

Tests of statistical significance were not performed. 

Program costs are based on a steady state of operation that excludes research and start-up costs. 

Credits attempted include all college-level and developmental credits attempted. 

a The condition of "marginal cost equal to zero" refers to the ability of existing college resources to absorb 
the additional credits attempted by the program group without incurring new costs to the college. Indirect cost 
under the condition when marginal cost equals zero represents the case where existing college resources can 
be leveraged to accommodate changes in credits attempted at no additional cost to the college. 

b The condition of "marginal cost equal to average cost" describes a condition where existing college 
resources are unable to absorb the additional credits attempted by the program group without incurring new 
cost to the college. Indirect cost under the condition "marginal cost equal to average cost" represents the case 
where existing college resources cannot be leveraged to accommodate changes in credits attempted, therefore 
incurring additional costs to the college. The additional costs to the college, or the marginal cost of the 
additional credits attempted, is approximated as the average cost per credit attempted at the institution. 

here as the “low-end” estimate). The low-end estimate would be reasonable if students attempt- 
ed more credits by enrolling in a large lecture class where some seats are currently empty. Such 
a circumstance will not always be realistic; hence, the analysis includes an estimate where 
marginal cost is equal to average cost. A marginal cost equal to the average cost represents the 
case where existing resources are unable to absorb the cost of additional credits attempted 
because existing resources are already fully leveraged, meaning the college will require addi- 
tional resources (referred to here as the “high-end” estimate). Since the analysis considers these 
two conditions, costs will be presented as ranges, representing the upper and lower bounds. 
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The third row of Table 5.1 shows the indirect cost of the program, or the cost associated 
with program group members’ attempting more credits than control group members. Program 
group members attempted more credits than their control group counterparts in the two years of 
follow-up. On average, each program group member attempted 33.2 credits by the end of the 
follow-up period; this is 1 .6 credits more than the average control group member attempted. 
Multiplying the additional 1.6 credits attempted by the corresponding cost per credit gives a 
high-end indirect cost of the program of $469 per program group member. The low-end indirect 
cost is $0, assuming that HCC can fully absorb the cost of additional credits attempted. 

Net Cost 

The net cost is defined as the difference between the program group cost and the control 
group cost. The costs of each group are presented in the total line of Table 5.1. Adding the 
direct cost, base cost, and indirect cost shows that the total cost of educating the average 
program group member over two years was between $10,353 and $10,822, while the total cost 
of educating the average control group member was $8,959. Over the two years of follow-up, 
the net cost can range from $1,394 to $1,863 per program group member. 


Cost-Effectiveness Analysis 

A cost-effectiveness analysis expresses the costs of alternative interventions as the cost per 
unit of a desired outcome. 9 This analysis considers the cost per college-level math course 
completion, the cost per math credit earned, and the cost per credit earned. Table 5.2 summa- 
rizes the results. The top row of Table 5.2 presents the total cost per group member. These 
values were described in the preceding “Net Cost” section of this chapter. The total cost per 
program group member ($10,353 to $10,822) is $1,394 to $1,863 more than the total cost per 
control group member ($8,959). The second row shows the percentage of program and 
control group members who completed college-level math. Specifically, 31.9 percent of 
program group members completed a college-level math course while 26.4 percent of control 
group members did so (as highlighted previously in Table 4.1). The next row presents the cost 
per college-level math completion for each group: $33,999 ($8,959/26.4 percent) per college- 
level math completion for the control group and $32,405 ($10,353/31.9 percent) to $33,873 
($10,822/31.9 percent) for the program group. The 5.6 percentage point increase in passing 


9 For additional explanation of this approach, see Sommo, Mayer, Rudd, and Cullinan (2012). A similar 
approach is also used in Levin and Garcia (2012). 
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The Performance-Based Scholarship Demonstration 
Table 5.2 

Cost-Effectiveness Values 
Hillsborough Community College 


Outcome 

Program 

Group 3 

Control 

Group 

Difference 

Cost per group member ($) 

10,353 to 10,822 

8,959 

1,394 to 1,863 

Passed college-level math (%) 

31.9 

26.4 

5.6 * 

Cost per passed college-level math ($) 

32,405 to 33,873 

33,999 

-1,594 to -126 

Math credits earned 

5.6 

4.9 

0 7 ** 

Cost per math credit earned ($) 

1,836 to 1,919 

1,826 

10 to 93 

Total credits earned 

23.1 

21.5 

1.6 

Cost per total credit earned ($) 

448 to 468 

416 

32 to 52 

Sample size (total = 1,075) 

674 

401 



SOURCES: MDRC calculations from program-specific participation and budget data, HCC transcript data, and 
Hillsborough Community College financial and enrollment data from Integrated Postsecondary Education Data 
System (http://nces.ed.gov/ipeds/datacenter/). 

NOTES: Rounding may cause slight discrepancies in cost calculations due to the rounding of impact data to one 
decimal place. All cost calculations used all four available decimal places in impact data for accuracy. All dollar 
values have been rounded to the nearest whole dollar. Estimates are adjusted by cohort-campus interaction, 
National Center for Education Statistics risk factors, and selected pre-random assignment placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Tests of statistical significance have been performed 
on outcome measures, not costs. All outcomes are for the first two years (cumulative). 

Program costs are based on a steady state of operation that excludes research and start-up costs. 
a Program group costs are shown as a range where the lower bound represents the point at which marginal cost 
equals zero, and the upper bound represents the point at which marginal cost equals average cost. 

college-level math was large enough to lower the cost per college-level math completion by up 
to 5 percent. 10 

A similar analysis is performed in relation to math credits earned and total credits 
earned. Cost-effectiveness results for math credits earned show that the program group cost per 
math credit earned is not lower than the cost per math credit earned without the program. 1 1 


10 Costs have been calculated with upper- and lower-bound conditions; therefore the upper-bound cost 
lowers the cost per college-level math completion by less than 1 percent while the lower-bound cost decreases 
the cost per college-level math completion by approximately 5 percent. 

11 The cost of math credits earned is based on the entire cost of attending college, not simply the cost of 
math courses. If this exercise were conducted based on the estimated cost of math courses only, the cost per 

(continued) 


74 






Likewise, the program did not lower the cost per total credit earned. 12 Specifically, the cost per 
credit earned for the program group ranges from $448 to $468 compared with the control 
group cost per credit earned of $416, as shown in the last two rows of Table 5.2, a difference 
of 8 to 12 percent. 

While the cost-effectiveness findings of MAPS are mixed, a college may opt to pay 
more for increased outcomes in order to achieve a greater rate of success at the expense of a 
higher cost per outcome. A college that places a high value on students’ earning more credits 
would still view such a program as a wise investment if the college were willing to pay more for 
students to succeed in earning more credits. 

In summary, the additional $1,394 to $1,863 invested in the program group members 
through the MAPS program yielded a 5.6 percentage point increase in the likelihood of com- 
pleting college-level math, a 0.7 increase in math credits earned, and a 1.6 increase in total 
credits earned over the perfonnance of the control group members. 


Conclusion 

The analysis shows that the $1,394 to $1,863 of additional investment in each program group 
member resulted in a 5.6 percentage point increase in the likelihood of completing a college- 
level math course. This impact is large enough that it lowers the cost per college-level math 
completion compared with the usual college services without the program. Specifically, the cost 
per college- level math completion for the program group is $126 to $1,594 (up to 5 percent) 
less than the cost per college-level math completion by the control group. This difference is 
important because it shows that while society did invest more resources in each program group 
member, the impact of the program on students’ completing college- level math was large 


math credit earned for the program group would still not be lower than the cost per math credit earned for the 
control group. 

12 The fact that MAPS lowered the cost per outcome in terms of completing college-level math but not in 
total credits earned highlights the importance of the relationship between the percentage increase in cost and 
the percentage change for an outcome. For example, under conditions where marginal cost is equal to average 
cost, the increase in costs associated with MAPS incurred over the normal college experience is 20.8 percent. 
Meanwhile, the 5.6 percentage point increase in completing college-level math by program group students is 
a 21.2 percent change over control group students who completed a college-level math course. This 21.2 
percent change is more than enough to account for the 20.8 percent increase in cost, and therefore MAPS 
lowers the cost per outcome with regard to the completion of a college-level math course. In the case of total 
credits, the total credits earned were 23.1 and 21.5 credits for the program group and control group, respec- 
tively. This impact of 1.6 credits is only a 7.4 percent change in total credits earned, not enough to overcome 
the 20.8 percent increase in costs incurred by MAPS. Therefore, MAPS does not lower the cost per outcome 
with regard to earning more total credits and similarly does not lower the cost per outcome in earning more 
math credits. 
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enough to lower the cost per outcome achieved compared with the usual college experience. 
When the cost is compared with other outcomes, such as math credits and total credits earned, 
the program is unable to lower the cost per outcome achieved. Depending on the value assigned 
to certain outcomes, such as math completion or total credits earned, producing a greater 
number of successful academic outcomes may be more important than lowering the cost per 
outcome. 

Understanding how much is paid to produce various outcomes can yield useful infor- 
mation for entities that are considering designing contracts for programs where payment is 
based on achieving specific outcomes. Outcomes are already implicitly purchased; however, it 
often is not known how much is being paid per outcome because costs are generally viewed in 
tenns of inputs, such as teacher instructional time. Cost-effectiveness analysis sheds light on 
how much is currently being spent to produce certain outcomes and how much specific pro- 
grams spend to produce the same outcomes. Knowledge about outcome prices provides a 
starting point for a conversation about what is an appropriate price to achieve a specific goal. 

The cost-effectiveness results show that HCC currently spends $33,999 per college- 
level math completion. 13 Since it already spends this much per college-level math completion, it 
may be reasonable for the college to pay a similar rate for increases in the number of college- 
level math completions. If HCC agreed to purchase increases in the number of college-level 
math course completions from MAPS at the same rate it currently pays, the payment would 
have been more than enough to cover the additional cost incurred due to MAPS. 14 


13 Value is based on the total cost of college for all students in the control group during the follow-up peri- 
od divided by the number of control students who passed college-level math. 

14 The way that colleges are funded is a barrier to this type of contracting. Current funding systems general- 
ly do not pay based on outcomes but, rather, based on inputs (such as teacher instructional time) or outputs 
(such as the number of credits attempted), and it is difficult to enter into contracts that pay based on outcomes 
when funding is not allocated based on outcomes. If funding streams were reformed based on the principle of 
paying for outcomes, contracts similar to the example above would be much more appealing in many settings 
because revenues could be expected to change as the amount of desired outcomes that are produced change. 
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Chapter 6 

Conclusion 


The Mathematics Access Performance Scholarship (MAPS) program at Hillsborough Commu- 
nity College (HCC) was implemented to help low-income students requiring developmental 
math get through the developmental course sequence and pass a college-level math course. 
MDRC’s evaluation of the program found that MAPS was successful at what it set out to do: 
move students further through the sequence. At the end of two years, program group students 
were more likely to have completed a college-level math course or Intermediate Algebra than 
were control group students. 

As shown in prior chapters of this report, MAPS was implemented with fidelity to the 
model. When surveyed, program group students reported a high level of awareness of the 
scholarship requirements and said that the scholarship was useful to their academic success. In 
focus groups, students said that the scholarship kept them a bit more focused on doing well in 
math. MAPS requirements also exposed more program group students to the supports available 
in Math Labs to help them leam course material. Many said they would not have visited the labs 
otherwise, meeting a critical goal of the MAPS developers at HCC. Indeed, data from Math Lab 
visits indicates that program group students were more likely to visit the Math Labs than control 
group students. 

While the program did not have a discernible effect on students’ persistence during the 
first two years, it did have a positive effect on math progress. The impact findings in this report 
indicate that MAPS met its primary goal for math progress, as it got more students to take and 
pass math courses and to continue on to enroll in the next course in the sequence. Students in 
the program group also earned more total credits than did students in the control group over the 
follow-up period. The cost-effectiveness analysis tells a more nuanced story that varies depend- 
ing on the primary outcome of interest. While MAPS lowered the cost per college-level math 
course completion, costs were higher per math credit and overall credit earned. 

The findings at HCC align with those at other colleges in MDRC’s Performance-Based 
Scholarship (PBS) Demonstration. By and large, the colleges in the demonstration have found 
that performance-based scholarships are able to improve some key academic outcomes, such as 
meeting the targeted academic benchmark and earning more credits. However, some academic 
outcomes remain mostly unaffected, such as student retention to the next semester. The findings 
at HCC, and across the demonstration, are modest but positive, indicating that performance- 
based scholarships can provide students with a small push in the right direction. 
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HCC’s Math Lab requirement may raise the question of whether adding a service com- 
ponent to a perfonnance-based scholarship is especially valuable for struggling students. The 
design of this study precludes the ability to separate the program’s components to determine the 
relative value of each. Indeed, none of the individual PBS studies were designed to answer the 
question of service components’ value. However, in a future report MDRC plans to use availa- 
ble data to look for lessons that may prove of use to the field on student service components. 


Changes to Developmental Education 

The results from this report indicate that MAPS can be a useful approach in affecting student 
behavior, though it does not fully solve the problem of students’ struggles with developmental 
education. The challenges of developmental education are complex, and programs that have 
small impacts seem to be the norm. Previous experimental evaluations conducted by MDRC 
have shown that developmental math learning communities , 1 for instance, produced small, 
positive impacts on developmental math credits attempted and earned, but no impact on 
persistence or total credits earned . 2 In fact, after a few semesters, control group students had 
largely caught up with program group students. 

Community colleges around the country continue to create and implement innovative 
strategies to improve student success in developmental math. Some, like MAPS, are interven- 
tions that take place completely outside the classroom , 3 while others change pedagogy, curricu- 
lum, or both. Many community colleges across the country have begun offering alternatives to 
the traditional lecture course, including courses based in computer labs using math module 
software and “flipped” courses in which students take a more active role in learning and 
presenting material in the classroom . 4 Two significant trends in developmental math curricular 
reform include accelerated courses, which shorten the timing or content of the courses so that 
students can get to college-level math more quickly, and modularized courses, which divide 
content into multiple separate modules, allowing students to speed through the material they 
already know so they can focus on the topics they don’t. Case studies of accelerated and 
modularized courses indicate some positive outcomes, although they have not yet been rigor- 
ously tested . 5 MDRC has just begun a random assignment evaluation of a modularized ap- 


'tn the learning community model, a cohort of students is coenrolled in two or more classes together, often 
with curricula and assignments that are thematically linked. 

2 Weissman et al. (2011). 

3 See, for example, the City University of New York’s (CUNY’s) Accelerated Study in Associate Pro- 
grams for developmental students, described in Scrivener and Weiss (2013). 

4 Strayer (2012). 

5 Bragg and Barnett (2009); Zachry (2008). 
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proach called ModMath at Tarrant County Community College in Texas as a part of the 
Developmental Math Acceleration project, seeking to evaluate the efficacy of this model 
compared with typical college math courses. 

A few proposals have taken more systemic approaches to changing developmental 
mathematics, considering classroom pedagogy, curriculum, supports to students, articulation, 
and course sequencing together. The idea is to improve the processes, strategies, and structures 
of developmental math instruction simultaneously. Notable approaches in this vein currently 
being piloted include CUNY Start, Statway/Quantway, and the New Mathways Project. 6 

At the state level, some proposals are taking on the developmental education challenge 
in completely new ways. In Florida, for example, the home of HCC, Senate Bill 1 720 requires 
that recent high school graduates may bypass both placement testing and developmental 
classwork if they so choose. As of January 2014, all of these students may progress directly to 
college-level courses. The diploma or certificate is presumed to indicate college-ready status. 
High school juniors in Florida are now required to take an assessment test that determines 
college readiness in reading, writing, and math. If they are assessed to need remediation, they 
must address this during their senior year of high school. Older students are still required to take 
placement tests, and the Florida Community Colleges must provide these students with both 
intensive advising and a variety of developmental instructional types, including some men- 
tioned above, such as co-requisite and modularized courses. 

Other states, such as Texas, plan to revamp the developmental education system, recast- 
ing the placement tests students take and referring some students who might have previously 
required developmental education to instead take Adult Basic Education courses before begin- 
ning community college. Research into the effects and effectiveness of legislative strategies like 
these is warranted as more states are considering or have enacted similar approaches to devel- 
opmental education. 


Conclusion and Next Steps 

Incentives and supports paired in a program like MAPS can be effective in improving rates of 
student success through a sequence of developmental mathematics courses. Although perfor- 
mance-based scholarships are not a complete solution to the developmental math question, this 
evaluation provides evidence that they can help low-income community college students’ 

6 For more information on CUNY Start, see www.cimy.edu/academics/programs/notablc/CATA/cti- 
cunystart.html. For more information on Statway/Quantway, see www.carnegiefoundation.org/statway. For 
more information on the New Mathways Project, see www.utdanacenter.org/higher-education/new-mathways- 
project. MDRC plans to evaluate the New Mathways Project in the coming years. 
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progress in math. While the study itself has ended, a small number of MAPS scholarships 
continue to be offered to students on all HCC campuses. HCC is hopeful that these scholarships 
can make a difference in students’ academic success. 

While the MAPS program is not as comprehensive as some of the developmental edu- 
cation overhauls discussed above, its impacts on student outcomes suggest that smaller, “outside 
of the classroom” programs can have a modest effect on students. For institutions unable to 
undertake significant overhauls, MAPS and interventions like it can be a first step or comple- 
ment to other programs targeting developmental education students. The MAPS findings also 
suggest the potential power of requirements. In this study, the performance-based scholarship 
provided the incentive for students to enroll in math, visit Math Labs, and ultimately pass their 
math courses at higher rates. One question worth considering is whether some of these effects 
could be produced by requiring students to take their developmental courses early and consecu- 
tively, or requiring Math Lab attendance (or its equivalent), or both. Requiring students to take 
developmental courses immediately is a policy change that some institutions are beginning to 
test. It is unclear how effective these strategies would be without the perfonnance-based 
scholarship as its financial benefits may extend beyond its motivational use; however, require- 
ments coupled with an incentive program like MAPS to get students into tutoring or learning 
assistance centers may be an interesting variant of a performance-based scholarship. 

A cross-site report on the PBS Demonstration will be published in 2015. This report 
will include longer-term follow-up for all sites, including HCC, and will look at patterns and 
impacts across sites to draw lessons from the larger demonstration. These findings will add to 
the growing body of knowledge about perfonnance-based scholarship models and the efficacy 
of incentives for improving academic success among low-income students. 
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The Performance-Based Scholarship Demonstration 
Appendix Table A.l 

Selected Baseline Characteristics, by Research Group 
Hillsborough Community College 

Program Group Control Group 

Standard Standard Estimated Difference 

Characteristic Mean Deviation Mean Deviation Mean P-value 


Gender (%) 


Male 

34.6 

47.6 

32.2 

46.8 

2.4 

0.4162 

Female 

65.4 

47.6 

67.8 

46.8 

-2.4 

0.4162 

Age (%) 

18-19 years old 

24.0 

42.8 

22.2 

41.6 

1.9 

0.4842 

20-23 years old 

23.1 

42.2 

24.5 

43.0 

-1.3 

0.6236 

24-26 years old 

13.4 

34.0 

12.2 

32.8 

1.2 

0.5850 

27-30 years old 

11.3 

31.7 

12.0 

32.5 

-0.7 

0.7095 

3 1 years and older 

28.2 

45.0 

29.2 

45.5 

-1.0 

0.7348 

Average age (years) 

27.1 

9.3 

26.8 

8.6 

0.2 

0.6933 

Race/ethnicity a (%) 

Hispanic 

30.9 

46.3 

30.3 

46.0 

0.7 

0.8136 

White 

32.1 

46.7 

27.9 

44.8 

4.2 

0.1459 

Black 

30.5 

46.1 

37.6 

48.5 

.7 i ** 

0.0179 

Asian or Pacific Islander 

1.7 

12.8 

0.5 

7.1 

1.2 * 

0.0973 

Other 

4.8 

21.4 

3.8 

19.1 

1.0 

0.4393 

Married (%) 

17.2 

37.8 

19.0 

39.2 

-1.8 

0.4679 

Missing 

8.6 

28.1 

11.0 

31.3 

-2.4 

0.1969 

Number of children (%) 

0 

57.3 

49.5 

54.2 

49.9 

3.0 

0.3378 

1 

17.8 

38.3 

16.7 

37.3 

1.1 

0.6474 

2 

12.1 

32.6 

13.1 

33.8 

-1.0 

0.6192 

3 or more 

12.8 

33.5 

15.9 

36.6 

-3.1 

0.1624 
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Appendix Table A.1 (continued) 


Characteristic 

Program Group 
Standard 
Mean Deviation 

Control Group 
Standard 
Mean Deviation 

Estimated Difference 
Mean P-value 

Among sample members with children 

Average age of youngest child (years) 

7.4 

6.2 

6.5 

5.7 



Disabilities or medical conditions' 3 (%) 

General learning disability (dyslexia, etc.) 

3.1 

17.4 

1.8 

13.1 

1.4 

0.1770 

Attention deficit/hyperactivity disorder, physical disability, or other 

9.6 

29.5 

6.7 

25.1 

2.9 

0.1001 

None of the above 

80.6 

39.6 

82.8 

37.8 

-2.2 

0.3693 

Missing 

8.2 

21 A 

9.2 

29.0 

-1.1 

0.5434 

Household receiving any government benefits c (%) 

37.7 

48.5 

38.5 

48.7 

-0.8 

0.7912 

Missing 

12.5 

33.1 

12.7 

33.4 

-0.2 

0.9073 

Financial dependence d (%) 

Independent 

59.2 

49.2 

63.1 

48.3 

-3.9 

0.2037 

Dependent 

36.2 

48.1 

30.9 

46.3 

5.3 * 

0.0765 

Missing 

4.6 

21.0 

6.0 

23.8 

-1.4 

0.3184 

Currently employed (%) 

50.3 

50.0 

51.5 

50.0 

-1.1 

0.7228 


Among those currently employed 

Number of hours worked per week in current job (%) 


1-10 hours 

4.6 

21.0 

3.0 

17.1 



11-20 hours 

24.1 

42.9 

23.7 

42.5 



21-30 hours 

22.7 

42.0 

19.5 

39.7 



31-40 hours 

46.3 

49.9 

48.6 

50.1 



More than 40 hours 

2.3 

14.5 

5.2 

22.9 



Average hourly wage at current job ($) 

10 

3 

11 

5 



Highest grade completed (%) 







1 0th grade or lower 

13.7 

34.4 

14.2 

35.0 

-0.6 

0.8009 

1 1th grade 

9.0 

28.6 

7.0 

25.5 

2.0 

0.2573 

12th grade 

11 A 

41.9 

78.8 

40.9 

-1.4 

0.5891 
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Appendix Table A.l (continued) 


Characteristic 

Program Group 
Standard 
Mean Deviation 

Control Group 
Standard 
Mean Deviation 

Estimated Difference 
Mean P -value 

Diplomas/degrees eamed b (%) 

High school diploma 

75.2 

43.2 

75.7 

42.9 

-0.5 

0.8419 

GED certificate 

26.9 

44.4 

25.3 

43.5 

1.6 

0.5597 

Occupational/technical certificate 

14.1 

34.8 

13.1 

33.8 

0.9 

0.6698 

Associate's degree or higher 

1.4 

11.5 

1.3 

11.2 

0.1 

0.8893 

None of the above 

0.1 

3.9 

0.0 

0.0 

0.1 

0.4500 

Date of high school graduation/GED receipt (%) 

During the past year 

18.7 

39.1 

18.2 

38.6 

0.6 

0.8154 

Between 1 and 5 years ago 

31.3 

46.4 

32.8 

47.0 

-1.4 

0.6272 

Between 5 and 10 years ago 

19.1 

39.3 

18.6 

39.0 

0.4 

0.8653 

More than 10 years ago 

30.9 

46.2 

30.4 

46.1 

0.4 

0.8822 

First semester at any college or university (%) 

24.6 

43.1 

22.2 

41.6 

2.4 

0.3721 

Expected enrollment in coming semester (%) 

Full time (12 credits or more) 

69.5 

46.1 

69.9 

45.9 

-0.4 

0.8952 

Part time (6 to 1 1 credits) 

27.9 

44.9 

28.9 

45.4 

-0.9 

0.7506 

Less than part time (less than 6 credits) 

2.6 

15.9 

1.3 

11.2 

1.3 

0.1554 

Main reason for enrolling in college 15 (%) 

To complete a certificate program 

2.1 

14.4 

1.5 

12.3 

0.6 

0.4924 

To obtain an associate's degree 

47.9 

50.0 

50.7 

50.1 

-2.7 

0.3898 

To transfer to a 4-year college/university 

48.0 

50.0 

46.8 

50.0 

1.2 

0.7069 

To obtain/update job skills 

3.3 

18.0 

1.8 

13.3 

1.5 

0.1420 

Other 

0.8 

8.7 

1.0 

10.1 

-0.3 

0.6456 

First person in family to attend college (%) 

33.4 

47.2 

33.1 

47.1 

0.3 

0.9231 
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Appendix Table A.l (continued) 


Characteristic 

Program Group 
Standard 
Mean Deviation 

Control Group 
Standard 
Mean Deviation 

Estimated Difference 
Mean P-value 

Highest degree/diploma earned by father (%) 

Not a high school graduate 

18.5 

38.9 

20.0 

40.0 

-1.4 

0.5663 

High school diploma or GED certificate 

31.7 

46.6 

32.4 

46.9 

-0.7 

0.8191 

Some college or associate's degree 

14.1 

34.8 

15.2 

36.0 

-1.2 

0.6026 

Bachelor's degree or higher 

11.4 

31.8 

5.2 

22.3 

6.2 *** 

0.0007 

Missing 

24.2 

42.9 

27.1 

44.5 

-2.9 

0.2876 

Highest degree/diploma earned by mother (%) 

Not a high school graduate 

16.8 

37.4 

14.7 

35.5 

2.1 

0.3699 

High school diploma or GED certificate 

36.4 

48.1 

34.1 

47.5 

2.2 

0.4605 

Some college or associate's degree 

23.9 

42.7 

28.2 

45.0 

-4.3 

0.1156 

Bachelor's degree or higher 

11.4 

31.8 

10.5 

30.7 

0.9 

0.6492 

Missing 

11.6 

32.0 

12.5 

33.1 

-0.9 

0.6666 

Language other than English spoken regularly in home (%) 

26.4 

44.1 

24.3 

43.0 

2.1 

0.4426 

Sample size (total = 1,075) 

674 


401 





SOURCE: MDRC calculations from Baseline Information Form data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort-campus interaction. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

Characteristics shown in italics are nonexperimental. Significance tests are not conducted for nonexperimental data. 

Missing values are included only in variable distributions for characteristics with more than 5 percent of the sample missing. 

Distributions may not add to 100 percent because of rounding. 

An omnibus f-test for joint significance was conducted for program and control groups. The test results were not significant at the 5 percent level, 
indicating that systematic differences between the program and control groups were not found. 

Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. Respondents who said they are not Hispanic and 
chose more than one race are included in the Other category. These respondents, combined with those who said they were American Indian, Alaska Native, 
or another race/ethnicity, are included in Other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 

c Benefits include unemployment/dislocated worker benefits, Supplemental Security Income or disability, cash assistance or welfare, food stamps, and 
Section 8 or public housing. 

d Students were counted as financially independent if they met any of the following criteria as of random assignment: they were age 24 years or older, they 
were married, or they had one child or more. 



Appendix B 

Survey Response Analysis 




This appendix discusses the student survey conducted as part of MDRC’s evaluation of the 
Mathematics Access Perfonnance Scholarship (MAPS) program. MDRC assessed the survey 
response rate and respondent characteristics to test the potential for bias in the result. This 
appendix also provides a supplementary survey table about students’ employment and 
motivation. 


Survey Fielding and Respondent Sample 

The student survey for the performance-based scholarship program at Hillsborough Commu- 
nity College asked study participants about a variety of topics, including their educational 
experiences, math and Math Lab experiences, experiences with MAPS (for program group 
students only), self-confidence, motivation, work experiences, financial situation, and psycho- 
logical well-being. The survey was fielded to the 1,075 sample members from March through 
May 2012, 1 during which time a total of 847 responses were collected. This was equivalent to 
an overall survey response rate of 79 percent. 2 


Characteristics of Survey Respondents 

Two different analyses for potential bias in survey responses were conducted. First, baseline 
characteristics of survey respondents were compared with the characteristics of students who 
did not respond to the survey, which provides an indication of how representative the survey 
respondents are of the full study sample — in other words, a form of external validity. Second, 
baseline characteristics for program group students who responded to the survey were compared 
with characteristics for control group students who responded to the survey; this provides an 
indication of whether the results are internally valid for survey respondents. 

Comparison of Respondent and Nonrespondent Characteristics 

Appendix Table B.l compares baseline characteristics for survey respondents and non- 
respondents. The table indicates that respondents and nonrespondents were similar with regard 


'Because the survey was fielded at the same time for all program cohorts, earlier cohorts had experienced 
more semesters of MAPS than later cohorts at the time of the fielding. In addition to the overall results 
presented here, survey results for each cohort were analyzed separately; results for individual cohorts were 
generally similar. 

2 Six students out of the study sample of 1,075 were excluded from the calculation of response rate. At the 
time of the survey fielding, these students were either deceased (1 student), away or unavailable to respond for 
the duration of the survey (4 students), or could not be contacted for other reasons ( 1 student). Excluding these 
6 students, a total of 1,069 students remained. The 847 responses represent 79 percent of this group. 
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The Performance-Based Scholarship Demonstration 
Appendix Table B.l 

Baseline Characteristics of Survey Respondents and Nonrespondents 
Hillsborough Community College 


Characteristic 

Respondents 

Non- 

respondents 

Difference 

Standard 

Error 

Gender (%) 

Male 

32.2 

39.1 

-6.9 * 

3.5 

Female 

67.8 

60.9 

6.9 * 

3.5 

Age (%) 

18-19 years old 

23.9 

21.1 

2.8 

3.2 

20-23 years old 

22.4 

28.2 

-5.8 * 

3.2 

24-26 years old 

12.3 

15.4 

-3.1 

2.5 

27-30 years old 

11.9 

10.3 

1.6 

2.4 

3 1 years and older 

29.5 

25.0 

4.5 

3.4 

Average age (years) 

27.1 

26.5 

0.6 

0.7 

Race/ ethnicity 3 (%) 

Hispanic 

31.4 

28.2 

3.2 

3.5 

White 

29.9 

32.7 

-2.8 

3.4 

Black 

33.1 

33.4 

-0.3 

3.6 

Asian or Pacific Islander 

1.3 

0.9 

0.5 

0.8 

Other 

4.3 

4.8 

-0.5 

1.6 

Married (%) 

18.6 

15.0 

3.6 

2.9 

Missing 

9.6 

9.0 

0.7 

2.2 

Number of children (%) 

0 

55.7 

57.8 

-2.2 

3.7 

1 

16.6 

20.3 

-3.7 

2.9 

2 

13.7 

8.0 

5 7 ** 

2.5 

3 or more 

14.0 

13.8 

0.2 

2.6 

Among sample members with children 

Average age of youngest child (years) 

7.2 

6.5 



Disabilities or medical conditions 1 * (%) 

General learning disability (dyslexia, etc.) 

2.6 

2.6 

0.0 

1.2 

Attention deficit/hyperactivity disorder, 

physical disability, or other 

8.3 

9.4 

-1.1 

2.1 

None of the above 

82.1 

78.8 

3.3 

2.9 

Missing 

8.0 

10.5 

-2.5 

2.1 

Household receiving any government 

benefits 0 (%) 

39.4 

32.6 

6.8 * 

3.6 

Missing 

12.6 

12.2 

0.4 

2.5 


(continued) 
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Appendix Table B.l (continued) 


Characteristic 

Respondents 

Non- 

respondents 

Difference 

Standard 

Error 

Financial dependence 11 





Independent 

60.9 

59.9 

1.0 

3.7 

Dependent 

33.9 

35.5 

-1.6 

3.6 

Missing 

5.3 

4.6 

0.6 

1.7 

Currently employed (%) 

50.4 

52.0 

-1.6 

3.8 

Among those currently employed 





Number of hours worked per week 





in current job (%) 





1-10 hours 

4.2 

3.4 



11-20 hours 

22.8 

28.1 



21-30 hours 

21.2 

22.4 



31-40 hours 

47.8 

44.7 



More than 40 hours 

4.0 

1.5 



Average hourly wage at current job ($) 

10.3 

10.5 



Highest grade completed (%) 





10 th grade or lower 

12.6 

18.5 

-5.9 ** 

2.6 

11th grade 

8.7 

6.5 

2.2 

2.1 

12 th grade 

78.7 

75.0 

3.7 

3.1 

Diplomas/degrees eamed b (%) 





High school diploma 

76.1 

72.7 

3.4 

3.2 

GED certificate 

25.5 

29.0 

-3.5 

3.3 

Occupational/technical certificate 

13.1 

16.1 

-3.0 

2.6 

Associate's degree or higher 

1.3 

1.3 

0.0 

0.9 

None of the above 

0.1 

0.0 

0.1 

0.2 

Date of high school graduation/GED receipt (%) 





During the past year 

18.7 

18.0 

0.7 

3.0 

Between 1 and 5 years ago 

31.6 

32.8 

-1.2 

3.6 

Between 5 and 10 years ago 

18.1 

22.1 

-4.0 

3.0 

More than 10 years ago 

31.6 

27.1 

4.5 

3.5 

First semester at any college or university (%) 

23.6 

24.2 

-0.7 

3.2 

Expected enrollment in coming semester (%) 





Full time (12 credits or more) 

70.6 

65.9 

4.7 

3.5 

Part time (6 to 11 credits) 

27.7 

30.5 

-2.8 

3.4 

Less than part time (less than 6 credits) 

1.7 

3.6 

-1.9 * 

1.1 

Main reason for enrolling in college 11 (%) 





To complete a certificate program 

1.9 

1.9 

0.0 

1.0 

To obtain an associate's degree 

50.0 

45.1 

4.8 

3.8 

To transfer to a 4-year college/university 

46.2 

52.4 

-6.1 

3.8 

To obtain/update job skills 

3.0 

1.8 

1.2 

1.2 

Other 

0.7 

1.4 

-0.7 

0.7 


(continued) 
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Appendix Table B.l (continued) 


Characteristic 

Respondents 

Non- 

respondents 

Difference 

Standard 

Error 

First person in family to attend college (%) 

33.6 

32.0 

1.6 

3.6 

Highest degree/diploma earned by father (%) 

Not a high school graduate 

19.6 

17.3 

2.3 

2.9 

High school diploma or GED certificate 

30.6 

37.3 

-6.7 * 

3.5 

Some college or associate's degree 

15.0 

12.7 

2.3 

2.6 

Bachelor's degree or higher 

10.1 

5.6 

4.5 ** 

2.2 

Missing 

24.8 

27.2 

-2.4 

3.3 

Highest degree/diploma earned by mother (%) 

Not a high school graduate 

16.5 

14.0 

2.6 

2.7 

High school diploma or GED certificate 

35.1 

37.1 

-2.0 

3.6 

Some college or associate's degree 

25.5 

25.3 

0.2 

3.3 

Bachelor's degree or higher 

11.1 

10.9 

0.2 

2.3 

Missing 

11.7 

12.7 

-1.0 

2.4 

Language other than English spoken regularly 

in home (%) 

27.5 

18.6 

g 9 *** 

3.3 

Sample size (total = 1,075) 

847 

228 




SOURCE: MDRC calculations from Baseline Information Form data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Characteristics shown in italics are nonexperimental. Significance tests are not conducted for nonexperimental 
data. 

Missing values are only included in variable distributions for characteristics with more than 5 percent of the 
sample missing. 

Distributions may not add to 100 percent because of rounding. 

a Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are included in the Other category. 
These respondents, combined with those who said they were American Indian, Alaska Native, or another 
race/ethnicity, are included in Other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 

c Benefits include unemployment/dislocated worker benefits, Supplemental Security Income or disability, cash 
assistance or welfare, food stamps, and Section 8 or public housing. 

d Students were counted as financially independent if they met any of the following criteria as of random 
assignment: they were age 24 years or older, they were married, or they had one child or more. 


to most of the characteristics measured. However, several statistically significant differences 
were also found. Women were more likely to respond, with approximately 68 percent of 
respondents being women, compared with only 6 1 percent of nonrespondents. Individuals from 
households receiving government benefits were more likely to respond to the survey (39 percent 
of respondents compared with 33 percent of nonrespondents), while students whose highest 
completed high school grade was 10th grade or lower were less likely to respond (13 percent of 
respondents compared with 19 percent of nonrespondents.) 


92 






Students who planned to enroll less than part-time in the upcoming semester were less 
likely to respond (2 percent of respondents versus 4 percent of nonrespondents). Students with a 
father whose highest degree earned was a high school diploma or General Educational Devel- 
opment (GED) certificate were less likely to respond to the survey, while students whose father 
had earned a bachelor’s degree or higher were more likely to respond the survey. Lastly, 
students who regularly spoke a language other than English at home were more likely to 
respond to the survey: 27.5 percent of respondents fell into this category, while only 19 percent 
of nonrespondents did. 

An omnibus F-test was conducted to determine whether students’ baseline characteris- 
tics were jointly predictive of responding to the survey. 3 The F-test yielded a p-value of 0.05, 
suggesting that respondents and nonrespondents differ in their baseline characteristics. This 
suggests that the survey results may not generalize to nonrespondents. 

Comparison of Program and Control Group Respondent Characteristics 

A slightly higher proportion of program group students responded to the survey (8 1 
percent) compared with control group students (74 percent.) Appendix Table B.2 compares 
baseline characteristics for respondents in the program and control groups to determine whether 
respondents’ characteristics differed between the two research groups. The table shows that the 
two groups were generally comparable, with survey respondents in the program group being 
slightly more likely to have one child (18 percent) than survey respondents in the control group 
(14 percent). Similarly, respondents in the program group were also more likely than respond- 
ents in the control group to have a father who had earned a bachelor’s degree or higher: 12 
percent of program group respondents met this criterion, while 6 percent of survey respondents 
in the control group did. Given how many baseline variables were explored, seeing a few 
modest differences in baseline characteristics is not unexpected. 

An omnibus F-test was conducted to determine whether survey respondents’ baseline 
characteristics were jointly predictive of student’s experimental status. The results were not 
statistically significant, indicating little evidence that the groups of respondents were systemati- 
cally different at the outset of the study. 

Appendix Table B.3 compares program and control group members’ employment sta- 
tus and motivation to complete their coursework, as discussed in Chapter 3. 


’Logistic regression was used for this analysis, where the outcome was whether a sample member re- 
sponded to the survey and the predictor variables were students’ baseline characteristics. 
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The Performance-Based Scholarship Demonstration 
Appendix Table B.2 

Baseline Characteristics of Survey Respondents, by Research Group 


Hillsborough Community College 


Characteristic 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Gender (%) 

Male 

32.6 

31.5 

1.1 

3.4 

Female 

67.4 

68.5 

-1.1 

3.4 

Age (%) 

18-19 years old 

24.0 

23.8 

0.2 

3.1 

20-23 years old 

22.4 

22.2 

0.2 

3.0 

24-26 years old 

12.8 

11.4 

1.4 

2.4 

27-30 years old 

11.6 

12.5 

-0.8 

2.3 

3 1 years and older 

29.2 

30.1 

-1.0 

3.3 

Average age (years) 

27.2 

27.0 

0.2 

0.7 

Race/ethnicity a (%) 

Hispanic 

30.7 

32.7 

-2.0 

3.4 

White 

31.1 

27.2 

3.9 

3.3 

Black 

31.5 

36.4 

-4.9 

3.4 

Asian or Pacific Islander 

1.9 

0.3 

1.5 * 

0.8 

Other 

4.8 

3.4 

1.4 

1.5 

Married (%) 

19.1 

17.8 

1.4 

2.8 

Missing 

9.3 

10.1 

-0.8 

2.1 

Number of children (%) 

0 

55.8 

55.4 

0.4 

3.6 

1 

18.2 

13.7 

4.5 * 

2.7 

2 

13.0 

14.9 

-1.9 

2.5 

3 or more 

13.0 

16.0 

-3.0 

2.5 

Among sample members with children 

Average age of youngest child (years) 

7.4 

6.7 



Disabilities or medical conditions 11 (%) 

General learning disability (dyslexia, etc.) 

3.3 

1.4 

1.9 * 

1.1 

Attention deficit/hyperactivity disorder, 

physical disability, or other 

9.1 

6.7 

2.4 

2.0 

None of the above 

81.2 

83.9 

-2.7 

2.8 

Missing 

8.0 

8.0 

0.0 

2.0 

Household receiving any government benefits c (%) 

39.1 

40.0 

-0.9 

3.5 

Missing 

12.6 

12.7 

-0.1 

2.4 


(continued) 
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Appendix Table B.2 (continued) 


Characteristic 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Financial dependence 11 





Independent 

60.5 

61.7 

-1.2 

3.5 

Dependent 

34.8 

32.2 

2.6 

3.4 

Missing 

4.7 

6.0 

-1.3 

1.6 

Currently employed (%) 

49.9 

51.3 

-1.4 

3.6 

Among those currently employed 





Number of hours worked per week in current job (%>) 





1-10 hours 

4.6 

3.4 



11-20 hours 

23.9 

20.4 



21-30 hours 

22.5 

18.7 



31-40 hours 

46.2 

51.0 



More than 40 hours 

2.8 

6.5 



Average hourly wage at current job ($) 

10.1 

10.7 



Highest grade completed (%) 





10th grade or lower 

12.9 

12.2 

0.7 

2.4 

11th grade 

9.2 

7.9 

1.2 

2.1 

12 th grade 

77.9 

79.9 

-2.0 

3.0 

Diplomas/degrees eamed b (%) 





High school diploma 

75.4 

77.2 

-1.8 

3.1 

GED certificate 

26.6 

23.8 

2.8 

3.2 

Occupational/technical certificate 

13.4 

12.6 

0.8 

2.4 

Associate's degree or higher 

1.3 

1.4 

-0.1 

0.8 

None of the above 

0.2 

0.0 

0.2 

0.2 

Date of high school graduation/GED receipt (%) 





During the past year 

18.4 

19.6 

-1.2 

2.8 

Between 1 and 5 years ago 

30.6 

32.9 

-2.3 

3.4 

Between 5 and 1 0 years ago 

18.9 

16.4 

2.5 

2.8 

More than 10 years ago 

32.1 

31.1 

1.0 

3.4 

First semester at any college or university (%) 

24.7 

22.0 

2.7 

3.0 

Expected enrollment in coming semester (%) 





Full time (12 credits or more) 

70.5 

71.2 

-0.7 

3.3 

Part time (6 to 11 credits) 

27.5 

27.8 

-0.3 

3.3 

Less than part time (less than 6 credits) 

2.1 

1.0 

1.0 

0.9 

Main reason for enrolling in college 11 (%) 





To complete a certificate program 

2.2 

1.4 

0.9 

1.0 

To obtain an associate's degree 

48.9 

51.7 

-2.8 

3.6 

To transfer to a 4-year college/university 

46.5 

45.9 

0.6 

3.6 

To obtain/update job skills 

3.5 

2.0 

1.5 

1.2 

Other 

0.6 

1.0 

-0.5 

0.6 


(continued) 
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Appendix Table B.2 (continued) 


Characteristic 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First person in family to attend college (%) 

33.3 

34.3 

-0.9 

3.5 

Highest degree/diploma earned by father (%) 

Not a high school graduate 

18.6 

21.5 

-2.9 

2.9 

High school diploma or GED certificate 

30.2 

31.2 

-1.0 

3.3 

Some college or associate's degree 

14.7 

15.5 

-0.7 

2.6 

Bachelor's degree or higher 

12.4 

5.7 

6.7 

*** 2 2 

Missing 

24.1 

26.2 

-2.1 

3.1 

Highest degree/diploma earned by mother (%) 

Not a high school graduate 

17.1 

15.4 

1.7 

2.7 

High school diploma or GED certificate 

35.2 

35.2 

-0.1 

3.4 

Some college or associate's degree 

24.6 

27.2 

-2.6 

3.1 

Bachelor's degree or higher 

11.5 

10.4 

1.0 

2.3 

Missing 

11.7 

11.8 

-0.1 

2.3 

Language other than English spoken regularly in home (%) 

28.2 

26.5 

1.7 

3.2 

Sample size (total = 847) 

549 

298 




SOURCE: MDRC calculations from Baseline Information Form data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Characteristics shown in italics are nonexperimental. Significance tests are not conducted for nonexperimental 
data. 

Missing values are included only in variable distributions for characteristics with more than 5 percent of the 
sample missing. 

Distributions may not add to 100 percent because of rounding. 

a Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are included in the Other category. 
These respondents, combined with those who said they were American Indian, Alaska Native, or another 
race/ethnicity, are included in Other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 
c Benefits include unemployment/dislocated worker benefits, Supplemental Security Income or disability, cash 
assistance or welfare, food stamps, and Section 8 or public housing. 

d Students were counted as financially independent if they met any of the following criteria as of random 
assignment: they were age 24 years or older, they were married, or they had one child or more. 
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The Performance-Based Scholarship Demonstration 
Appendix Table B.3 
Student Employment and Motivation 


Hillsborough Community College 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Employment 

Currently working for pay or profit (%) 

71.3 

71.7 

-0.4 

3.0 

Number of hours usually worked per week 

22.9 

23.1 

-0.3 

1.2 

Identify as being "financially stressed" (%) 

Strongly agree 

37.9 

37.5 

0.4 

3.6 

Agree 

37.2 

35.8 

1.4 

3.6 

Disagree 

18.9 

20.3 

-1.3 

2.9 

Strongly disagree 

6.0 

6.5 

-0.5 

1.8 

Motivation 

Motivation to complete coursework (average) 

Relative autonomy index 3 

4.1 

3.8 

0.3 

0.3 

Sample size (total = 847) 

549 

298 




SOURCE: MDRC calculations using responses from the Performance-Based Scholarship Hillsborough 
Community College survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by 
cohort-campus interaction. National Center for Education Statistics risk factors, and selected pre- 
random assignment placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Missing values are not included in individual variable distributions. 

a The Relative Autonomy Index (RA1) has a possible range of -18 to 18, where a higher value 
represents greater autonomous motivation. The RA1 is calculated as a weighted average: RAI = 
[External x (-2)] + [Introjected x (-1)] + [Identified x (1)] + [Integrated x (2)]. 
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Appendix C 

Impact Tables 




The Performance-Based Scholarship Demonstration 
Appendix Table C.l 

Credits Attempted and Earned (Cumulative) 


Hillsborough Community College 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 


P-value 

First semester 

Total credits attempted 

10.7 

3.8 

10.5 

4.0 

0.3 


0.1923 

Math 

3.7 

1.0 

3.3 

1.5 

0.4 

*** 

0.0000 

Non-math 

7.0 

3.4 

7.1 

3.4 

-0.1 


0.4481 

Total credits earned 

7.5 

4.7 

6.8 

4.6 

0.7 

** 

0.0129 

Math 

2.3 

2.0 

1.8 

2.0 

0.4 

*** 

0.0010 

Non-math 

5.3 

3.7 

5.0 

3.6 

0.3 


0.1733 

First vear 

Total credits attempted 

21.6 

9.5 

20.7 

9.2 

0.9 


0.1021 

Math 

6.8 

2.6 

5.9 

2.7 

0.9 

*** 

0.0000 

Non-math 

14.8 

7.9 

14.8 

7.8 

0.0 


0.9517 

Total credits earned 

15.0 

10.5 

13.7 

10.0 

1.3 

** 

0.0451 

Math 

4.1 

3.4 

3.5 

3.3 

0.7 

*** 

0.0012 

Non-math 

10.9 

8.3 

10.3 

8.0 

0.6 


0.2280 

First vear and one semester 

Total credits attempted 

27.6 

13.5 

26.3 

13.0 

1.4 

* 

0.0926 

Math 

8.4 

3.4 

7.3 

3.6 

1.1 

*** 

0.0000 

Non-math 

19.3 

11.2 

19.0 

10.8 

0.3 


0.6790 

Total credits earned 

19.2 

14.3 

17.6 

13.6 

1.5 

* 

0.0845 

Math 

5.0 

4.2 

4.3 

4.0 

0.8 

*** 

0.0038 

Non-math 

14.1 

11.4 

13.4 

10.9 

0.8 


0.2686 

First two vears 

Total credits attempted 

33.2 

18.2 

31.6 

17.5 

1.7 


0.1367 

Math 

9.5 

4.4 

8.5 

4.6 

1.0 

*** 

0.0003 

Non-math 

23.7 

15.0 

23.1 

14.3 

0.6 


0.4994 

Total credits earned 

23.1 

18.1 

21.5 

17.6 

1.6 


0.1603 

Math 

5.6 

4.7 

4.9 

4.7 

0.7 

** 

0.0137 

Non-math 

17.5 

14.7 

16.6 

14.3 

0.9 


0.3463 

Sample size (total = 1,075) 

674 


401 






SOURCE: MDRC calculations from Hillsborough Community College transcript data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre-random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The Performance-Based Scholarship Demonstration 
Appendix Table C.2 

Credits Attempted and Earned (Noncumulative) 
Hillsborough Community College 


Program Group Control Group Estimated Effects 


Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

P -value 

First semester 

Enrolled (%) 

95.8 

20.3 

93.9 

23.8 

1.9 

0.1702 

Total credits attempted 

11.8 

4.8 

11.5 

5.1 

0.3 

0.2708 

Math 

4.1 

1.5 

3.6 

1.8 

q 5 *** 

0.0000 

Non-math 

7.7 

4.0 

7.9 

4.3 

-0.2 

0.3715 

Total credits earned 

8.3 

5.6 

7.7 

5.8 

0.6 * 

0.0747 

Math 

2.5 

2.3 

2.1 

2.2 

Q 5 *** 

0.0015 

Non-math 

5.8 

4.3 

5.6 

4.5 

0.2 

0.5445 

Second semester 

Enrolled (%) 

79.5 

40.4 

78.1 

41.4 

1.3 

0.6046 

Total credits attempted 

9.8 

7.0 

9.2 

6.5 

0.6 

0.1299 

Math 

2.7 

2.1 

2.3 

2.0 

Q ^ *** 

0.0017 

Non-math 

7.1 

5.7 

6.9 

5.4 

0.2 

0.5124 

Total credits earned 

6.8 

6.5 

6.2 

6.2 

0.6 

0.1088 

Math 

1.6 

2.0 

1.4 

2.0 

0.2 * 

0.0559 

Non-math 

5.2 

5.3 

4.8 

5.1 

0.4 

0.2170 

Third semester 

Enrolled (%) 

60.2 

49.1 

57.9 

49.3 

2.4 

0.4481 

Total credits attempted 

6.5 

6.3 

6.1 

6.0 

0.4 

0.2791 

Math 

1.6 

1.8 

1.5 

1.7 

0.2 

0.1398 

Non-math 

4.9 

5.2 

4.6 

4.9 

0.3 

0.4288 

Total credits earned 

4.6 

5.5 

4.4 

5.3 

0.2 

0.5381 

Math 

1.0 

1.6 

0.9 

1.5 

0.1 

0.3558 

Non-math 

3.6 

4.7 

3.5 

4.4 

0.1 

0.6779 


(continued) 
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Appendix Table C.2 (continued) 


Outcome 

Program Group 

Control Group 

Estimated Effects 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

P-value 

Fourth semester 







Enrolled (%) 

49.6 

50.0 

46.8 

50.0 

2.8 

0.3726 

Total credits attempted 

5.1 

6.1 

4.8 

6.0 

0.3 

0.4045 

Math 

1.1 

1.7 

1.1 

1.8 

0.0 

0.9468 

Non-math 

4.0 

5.2 

3.7 

4.9 

0.3 

0.3088 

Total credits earned 

3.8 

5.2 

3.7 

5.1 

0.1 

0.6730 

Math 

0.6 

1.3 

0.6 

1.4 

0.0 

0.9372 

Non-math 

3.2 

4.6 

3.1 

4.4 

0.1 

0.6088 

Sample size (total = 1,075) 

674 


401 





SOURCE: MDRC calculations from Hillsborough Community College transcript data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort- 
campus interaction, National Center for Education Statistics risk factors, and selected pre -random assignment 
placement test scores. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Note that spring and summer semesters are combined in the semesters shown here; see Figure 3.1 for more 
information. 
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Appendix D 

Key Cost Terms 




The following terms are used in the cost analysis, presented in Chapter 5, of the Mathematics 
Access Perfonnance Scholarship (MAPS) program at Hillsborough Community College as part 
of the Perfonnance-Based Scholarship Demonstration. 

Direct cost: the cost directly associated with providing performance-based scholarships and the 
program’s components. 

Base cost: the cost of the usual college services in the absence of the program. Base cost = cost 
per credit x number of credits attempted by the control group. The cost per credit is an estimate 
of the average amount of resources expended by the college to provide one credit’s worth of 
instructional activity; it is calculated by dividing the college’s annual operating budget by the 
number of credits attempted at the college during the year of interest. 

Indirect cost: the cost resulting from a behavioral change brought about by the program, such 
as additional credits attempted by program group members; such costs can extend beyond the 
period of program operation. Indirect cost of the program = cost per credit x additional credits 
attempted by program group members. 

Program group cost: the total cost of educating program group members over two years of 
follow-up. Program group cost = base cost + direct cost + indirect cost. Program group cost 
can be divided by the number of program group members to get the cost per program group 
member. 

Control group cost: the total cost of educating control group members over two years of 
follow-up. Control group cost = base cost. Control group cost can be divided by the number of 
control group members to get the cost per control group member. 

Net cost: the cost difference between program group members and control group members. Net 
cost = program group cost - control group cost. 

Cost-effectiveness analysis: an evaluation in which the net costs of alternative interventions are 
expressed as the cost per unit of a desired outcome. In this analysis, cost-effectiveness is 
presented for cost per college-level math completion, cost per math credit earned, and cost per 
toted credit earned. 

Cost per college-level math completion: the amount invested in the research group of interest 
per college-level math completion by that research group. For the program group, cost per 
college-level math completion = program group cost + number of college-level math comple- 
tions by program group members. 
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Cost per math credit earned: the amount invested in the research group of interest per math 
credit earned by that research group. For the program group, cost per math credit earned = 
program group cost + number of math credits earned by program group members. 

Cost per total credits earned: the amount invested in the research group of interest per total 
credit earned by that research group. For the program group, cost per total credits earned = 
program group cost 4 number of total credits earned by the program group members. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



